acceptodds
Under review as a conference paper at ICLR 2027

AQA: Anytime Action Quality Assessment via Flow-Based Score Forecasting

Abstract

Action Quality Assessment (AQA) typically predicts a final performance score only after observing the complete video. However, in sports and skill training, predicting performance outcomes before an action concludes can enable earlier and more prospective feedback. We study the anytime AQA task in this paper, where the goal is to predict the final score from a partially observed performance while accounting for uncertainty in its unseen future. We propose A2QA, a model-agnostic framework that augments a pretrained AQA model with plausible future completions. Given an observed prefix, A2QA retrieves real future segments from training videos with compatible declared content and appends them to the prefix in feature space. To model uncertainty over possible futures, we introduce a conditional flow model that estimates the likelihood of each retrieved completion conditioned on the observed prefix. A frozen scoring model then evaluates each completed sequence, and their likelihood-weighted scores are aggregated to predict the final performance score. The framework requires neither retraining nor modification of the underlying scoring model and naturally reduces to the original model when the full performance is observed. Experiments on four datasets spanning sports and surgical skill assessment (FineFS, FineDiving, JIGSAWS, and LapEx) show consistent improvements across all 13 model. For example, in TSMP on FineFS datasets, Spearman correlation improves from 0.395 to 0.661 at only 10% observation. These results demonstrate that explicitly modeling plausible future completions substantially improves final-score prediction from partial observations while preserving compatibility with existing AQA models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action Quality Assessment, Anytime Prediction, Partial Video Completion, Flow Matching, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_J9w6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A²QA predicts the final action quality score from a partial video. It appends compatible endings from training videos to the observed prefix in feature space, scores each completion with a frozen AQA model, and averages the scores using weights derived from conditional flow densities. With the full video available, it returns the original model’s prediction. Across four datasets and 13 model–dataset pairs, A²QA outperforms direct scoring of truncated videos at 10%, 30%, and 50% observation. FineDiving results also show lower numerical scoring errors.

Critical strengths
  1. A²QA reuses a pretrained full-video scorer while learning how to weight alternative real endings. This avoids requiring a new scoring architecture for every backbone and avoids synthesizing future video. Returning the original prediction at full observation also gives the framework a clear relationship to conventional AQA.

  2. Table 1 reports improvements at 10%, 30%, and 50% observation for all 13 model–dataset pairs. For example, TSMP on FineFS improves from 0.395 to 0.661 in Spearman correlation at 10% observation. FineDiving results include numerical score errors as well as ranking correlation. Table 3 further shows that learned density weighting improves over uniform and prefix-similarity weighting on skating and diving. These results support the effectiveness of the approach under the tested conditions.

Critical weaknesses
  1. The main baseline scores truncated videos with a model trained on complete videos. This comparison shows the benefit of adapting a frozen scorer, but does not establish an advantage over models trained for early prediction. Despite discussing prior partial-observation scoring work, the paper includes no such baseline. A comparison with a model trained on prefixes using final-score supervision should match information access and report training and inference costs.

  2. The method depends on compatible categories, declared content, and training-library coverage. Joining segments from different performances may produce implausible transitions, whose effect on the scorer is not tested. Low completion-score disagreement may also hide large errors when plausible futures are missing; Figure 6 does not establish calibration or reliable prediction withholding. Inference evaluates up to 30 candidates through density integration and scoring, but end-to-end latency is not reported.

Questions for authors
  1. How does A²QA compare with a predictor trained on randomly truncated videos using final-score supervision, and with applicable partial-observation AQA methods discussed in Section 2? Please match the observed inputs and access to action/declaration information, and report differences in training requirements and inference cost. This would clarify the trade-off between reusing a frozen scorer and training specifically for early prediction.

  2. How does performance change as the number or diversity of available candidate endings decreases, or when the action declaration is unavailable or incorrect? What happens if there is no compatible candidate? Can you provide evidence that incompatible transitions between the observed prefix and a retrieved ending do not systematically mislead the frozen scorer?

4 comments

We thank the reviewer for the positive assessment of the paper’s clarity, our reuse of pretrained scorers, and the consistent improvements across models and datasets. We address each question below.

Q1 · Comparison with trained partial-observation methods

Response:

Table R1 shows higher ρ for A²QA than fine-tuning (FT) in 38/39 settings and intermediate supervision (IS) in 39/39 settings at 10–50% observation.

Both training and inference for A²QA, FT, and IS can be performed on a single RTX 3090.

FT/IS train a predictor on partial videos. A²QA keeps the scorer frozen and trains a flow model. Its inference adds retrieval, density estimation, and scoring of up to 30 completions. Table R2 reports the encoding, density-estimation, and scoring times.

(to be continued)

Table R1. ρ and R-ℓ₂ × 100 at 10%, 30%, and 50% observation.

Spearman’s ρ (↑)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
SkatingTSMP0.3950.5130.5740.7590.3950.6610.6100.5750.5800.7590.6100.7770.5940.6170.6720.7590.5940.795
GDLT0.5790.6340.5740.7590.5790.7960.5890.6930.5800.7590.5890.8060.7230.7710.6720.7590.7230.805
B2S0.7680.7760.5740.7590.7680.8050.8270.8280.5800.7590.8270.8420.8480.8390.6720.7590.8480.857
DivingTSA0.5530.5860.4010.5880.5530.6120.5570.5970.5100.5880.5570.6380.5730.6150.5560.5880.5730.635
FineCausal0.5440.5570.4010.5880.5440.6140.5570.6020.5100.5880.5570.6400.5550.6130.5560.5880.5550.646
HP-MCoRe0.5790.5660.4010.5880.5790.6090.5860.5970.5100.5880.5860.6300.5940.6150.5560.5880.5940.635
CoRe0.5490.5750.4010.5880.5490.6000.5730.6060.5100.5880.5730.6110.5820.6130.5560.5880.5820.634
RICA²0.5870.5900.4010.5880.5870.6240.5960.6030.5100.5880.5960.6480.5860.5950.5560.5880.5860.651
Uni-FineParser0.5680.5900.4010.5880.5680.6040.5910.6290.5100.5880.5910.6250.6140.6420.5560.5880.6140.645
SurgeryCoRe0.3960.4230.218−0.2140.5190.6270.6580.6790.259−0.2140.7020.6960.6830.6700.359−0.2140.6970.720
RICA²0.4140.3770.218−0.2140.3770.5960.5640.5510.259−0.2140.5560.5750.6390.6240.359−0.2140.6160.676
LapExMoAKE0.1710.1200.0070.2240.1710.1990.1900.0930.0480.2240.1900.2440.2400.1270.1460.2240.2400.274
LIMSSR0.0560.0810.0070.2240.0560.1170.1030.1360.0480.2240.1030.1790.1370.1940.1460.2240.1370.232

R-ℓ₂ × 100 (↓)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
DivingTSA4.731.362.971.341.381.304.491.342.041.341.381.274.111.311.721.341.361.26
FineCausal2.131.352.971.341.351.232.001.282.041.341.351.212.051.251.721.341.341.18
HP-MCoRe3.031.452.971.341.341.302.551.372.041.341.321.251.961.321.721.341.301.22
CoRe1.851.642.971.341.391.261.771.552.041.341.351.231.811.601.721.341.331.19
RICA²1.531.312.971.341.291.211.571.322.041.341.271.161.651.381.721.341.291.15
Uni-FineParser1.951.342.971.341.361.311.861.292.041.341.331.261.801.291.721.341.291.21

FT: fine-tuning on truncated inputs; IS: intermediate supervision; Prior: zero-video group mean; Recal.: linear recalibration. Bold indicates the best value for each model and observation ratio.

(to be continued)

Table R2. Inference latency (ms) on one NVIDIA RTX 3090. Full = Encode + Density + Scorer.

DatasetScorerTrunc.EncodeDensityScorer (parallel)Full
FineFSTSMP34.550222634.3762.3
GDLT2.95102262.9738.9
B2S31.451122648.6785.6
FineDivingTSA24.76.628.719.254.5
FineCausal1,38017.426.11,3881,431.5
HP-MCoRe76.36.628.776.3111.6
CoRe8.012.630.51.444.5
RICA²25.213.228.725.267.1
Uni-FineParser16.921.438.31.461.1
JIGSAWSCoRe43.66.130.71.738.5
RICA²1,08811.630.71,0801,122.3
LapExMoAKE12.55.732.912.551.1
LIMSSR53.75.732.953.792.3

Q2 · Candidate availability, declarations, and transition robustness

Response:

Candidate number and diversity. On FineFS/TSMP at 10% observation, reducing the candidate count from 30 to 3 or 1 lowers mean ρ from 0.661 to 0.527 or 0.392. With 15 candidates, a less diverse set of similar endings gives ρ of 0.606, versus 0.641 for random selection.

Missing or incorrect declarations. To test missing declarations, we retrieve candidates from all training videos using prefix similarity. For TSMP on FineFS at 10% observation, ρ decreases from 0.661 with declarations to 0.513 without them. Giving TSMP the wrong label lowers ρ at 10% observation from 0.661 to −0.268.

No compatible candidate. If the pool is empty, the method cannot produce a completion-based score; handling this case remains future work.

Transitions at the join. On FineFS/TSMP at 10% observation, smoothing one or four feature rows at the join gives ρ of 0.656 or 0.643, versus 0.661 before smoothing and 0.395 for Trunc. The ranking advantage remains under these local changes.

We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.

The added comparisons with fine-tuning and intermediate supervision strengthen the evidence for A²QA, while the latency measurements and robustness experiments address several of my concerns. However, equivalent access to action and declaration information across baselines remains unclear, and the method still depends strongly on candidate coverage and correct declarations. The transition and uncertainty analyses provide useful but limited evidence of reliability. I therefore maintain my rating of 3 (Weak acceptance).

Reviewer_i7s1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

For a partially observed video, A2QA retrieves up to 30 real endings from training performances of the same category and declaration, appends them to the observed prefix in feature space, scores each completed sequence with a frozen AQA model, and averages those scores with weights from a conditional flow-matching density over the ending (with a reference correction and a temperature fitted on training data). The weighted standard deviation is reported as uncertainty, and at full observation the framework returns the original model's output. Four datasets, eleven frozen scorers, thirteen model–dataset pairs.

Critical strengths

First, the problem is worth studying. Retrospective-only scoring is a real limitation, and returning the original prediction at x = 1 is a clean design property. Second, the method is training-light and genuinely model-agnostic: eleven scorers from different families are reused without modification, which is more coverage than most AQA papers provide. Last, the empirical study is broad — four datasets, five observation ratios, two metrics — and the flow-versus-diffusion comparison is reported even though it is unflattering.

Critical weaknesses
  1. The only baseline is too weak to support the claims. It is the same frozen model applied to a truncated video, i.e. an input distribution the model was never trained on. Three baselines are needed: (a) the same scorer fine-tuned or retrained with truncation augmentation, or with Parmar–Morris-style intermediate supervision, which exists for exactly this purpose and is cited by the authors; (b) a zero-video predictor that returns the mean training score of the same category/declaration group; (c) a linear recalibration of the truncated model's outputs, fitted on held-out partial inputs. Without (a) and (b) the paper shows only that complete-video models are bad at scoring incomplete videos, which is not in dispute.
  2. The ablation undercuts the paper's main technical component. In Table 3(a), cosine-similarity weights reach mean ρ 0.689 on diving against 0.690 for the full method, and the reference correction contributes +0.014. In Table 2, flow matching and three diffusion variants lie within 0.01 of each other on skating and diving. The claimed benefit of "conditional density modelling over candidate futures" is therefore not visible in the authors' own numbers; a simple similarity kernel reproduces it. The one setting where density weighting does help (skating, +0.09) is also the setting where the reference pool is built by retrieving the ending, which needs explanation rather than a table entry.
  3. Predictions barely respond to the observation ratio. For TSA, ρ = 0.612, 0.638, 0.635, 0.747, 0.899 and R-ℓ² = 1.30, 1.27, 1.26, 0.95, 0.45 at 10/30/50/70/90%. Between 10% and 50% the prediction is essentially unchanged, which is the opposite of what a method based on "the space of plausible futures becomes increasingly constrained" should do. The diagnostic that would settle this is missing: the correlation between ŷ(x) and the mean gold score of the retrieved candidates, and the variation of ŷ(x) across donors.
  4. Construct validity: the method re-scores a chimera rather than forecasting a future. The frozen scorer receives [query prefix ; donor ending], so its output is dominated by the donor's execution. The paper should report the correlation between ŷ(x) and the donor's own gold score; if it is high, the honest description is retrieval-based score transfer, which may still be useful but is not the claim being made.
  5. The reported uncertainty is never evaluated. σ(x) is proposed as the basis for withholding unreliable predictions, but there is no calibration curve, no risk–coverage analysis, and no correlation between σ and absolute error. As presented it is decorative.
  6. The task is not new, and there is no limitations section. Predicting final scores from partial video is prior work cited in the paper itself (Parmar & Morris 2017; Gazis et al. 2026; Xu et al. 2015). The paper should position itself as a framework for extending complete-video models, not as a new task. Behaviour when no compatible ending exists, the cost of scoring thirty completions per query, and the difference between sports (where the declaration fixes much of the score) and surgery are not discussed.
Questions for authors
  1. With the same scorer fine-tuned on truncated inputs, or with intermediate-score supervision, what is ρ at 10–50%, and does A2QA still help?
  2. What is the correlation between ŷ(x) and the donor ending's gold score, and between ŷ(x) and the pool mean? What is ρ for a zero-video group-mean predictor on the same declaration?
  3. Why does density weighting gain 0.09 on skating but 0.001 on diving, and why is the skating reference pool constructed by retrieval?
  4. What is the empirical relation between σ(x) and absolute error, and what risk–coverage trade-off does withholding high-σ predictions produce?
  5. Are the gains stable across truncation offsets and seeds, given that the truncated baseline is non-monotonic in x (TSMP: 0.395, 0.610, 0.594, 0.532, 0.631)?
4 comments

We thank the reviewer for the positive assessment of the problem’s value, our model-agnostic design, and the broad experimental study. We address each question below.

Q1 · Fine-tuned scorer and intermediate supervision

Response:

Table R1 reports ρ at 10%, 30%, and 50% observation. Across 13 model–dataset pairs, A²QA outperforms fine-tuning on truncated inputs (FT) in 38 of 39 settings and intermediate supervision (IS) in all 39 settings, while keeping the original scorer frozen.

Table R1. ρ and R-ℓ₂ × 100 at 10%, 30%, and 50% observation.

Spearman’s ρ (↑)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
SkatingTSMP0.3950.5130.5740.7590.3950.6610.6100.5750.5800.7590.6100.7770.5940.6170.6720.7590.5940.795
GDLT0.5790.6340.5740.7590.5790.7960.5890.6930.5800.7590.5890.8060.7230.7710.6720.7590.7230.805
B2S0.7680.7760.5740.7590.7680.8050.8270.8280.5800.7590.8270.8420.8480.8390.6720.7590.8480.857
DivingTSA0.5530.5860.4010.5880.5530.6120.5570.5970.5100.5880.5570.6380.5730.6150.5560.5880.5730.635
FineCausal0.5440.5570.4010.5880.5440.6140.5570.6020.5100.5880.5570.6400.5550.6130.5560.5880.5550.646
HP-MCoRe0.5790.5660.4010.5880.5790.6090.5860.5970.5100.5880.5860.6300.5940.6150.5560.5880.5940.635
CoRe0.5490.5750.4010.5880.5490.6000.5730.6060.5100.5880.5730.6110.5820.6130.5560.5880.5820.634
RICA²0.5870.5900.4010.5880.5870.6240.5960.6030.5100.5880.5960.6480.5860.5950.5560.5880.5860.651
Uni-FineParser0.5680.5900.4010.5880.5680.6040.5910.6290.5100.5880.5910.6250.6140.6420.5560.5880.6140.645
SurgeryCoRe0.3960.4230.218−0.2140.5190.6270.6580.6790.259−0.2140.7020.6960.6830.6700.359−0.2140.6970.720
RICA²0.4140.3770.218−0.2140.3770.5960.5640.5510.259−0.2140.5560.5750.6390.6240.359−0.2140.6160.676
LapExMoAKE0.1710.1200.0070.2240.1710.1990.1900.0930.0480.2240.1900.2440.2400.1270.1460.2240.2400.274
LIMSSR0.0560.0810.0070.2240.0560.1170.1030.1360.0480.2240.1030.1790.1370.1940.1460.2240.1370.232

R-ℓ₂ × 100 (↓)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
DivingTSA4.731.362.971.341.381.304.491.342.041.341.381.274.111.311.721.341.361.26
FineCausal2.131.352.971.341.351.232.001.282.041.341.351.212.051.251.721.341.341.18
HP-MCoRe3.031.452.971.341.341.302.551.372.041.341.321.251.961.321.721.341.301.22
CoRe1.851.642.971.341.391.261.771.552.041.341.351.231.811.601.721.341.331.19
RICA²1.531.312.971.341.291.211.571.322.041.341.271.161.651.381.721.341.291.15
Uni-FineParser1.951.342.971.341.361.311.861.292.041.341.331.261.801.291.721.341.291.21

FT: fine-tuning on truncated inputs; IS: intermediate supervision; Prior: zero-video group mean; Recal.: linear recalibration. Bold indicates the best value for each model and observation ratio.

(to be continued)

Q2 · Donor correlation and zero-video group-mean predictor

Response:

We report Spearman correlations between ŷ and the flow-weighted or unweighted mean of donors' full-video gold scores.

The zero-video group-mean predictor uses the mean training score of the same declaration group, without video input. We report Spearman ρ against the test videos' true scores.

DatasetZero-video ρ
FineDiving0.588
FineFS0.759
JIGSAWS−0.214
LapEx0.224

For donor sensitivity, we keep each observed prefix fixed and resample the same number of donors from its original pool 1,000 times, allowing repeats and renormalizing the flow weights. SD measures the variation in the final prediction across these draws.

DatasetObservation ratioρ(ŷ, weighted donor gold mean)ρ(ŷ, unweighted pool mean)Mean SD of final ŷ
FineDiving10%0.9860.9862.143
30%0.9820.9662.228
50%0.9800.9732.130
70%0.9620.9451.816
90%0.9250.9000.688
FineFS10%0.9390.9011.593
30%0.8040.7131.229
50%0.7890.7271.016
70%0.7010.6140.618
90%0.6500.6580.456
JIGSAWS10%0.892−0.2291.041
30%0.350−0.2570.762
50%0.359−0.2630.576
70%0.542−0.2570.357
90%0.592−0.2450.142
LapEx10%0.8010.7250.045
30%0.6410.6600.021
50%0.5380.5750.015
70%0.4650.4960.009
90%0.3450.3680.005

Donor diagnostics average all scorers and available seeds.

These results support reduced sensitivity to candidate endings as more video is observed. From 10% to 90%, weighted donor-score correlation falls from 0.939 to 0.650 on FineFS and 0.801 to 0.345 on LapEx. Mean SD decreases by 68–88% across all four datasets: at every evaluated ratio on FineFS, JIGSAWS, and LapEx, and mainly at later stages on FineDiving.

ρ does not necessarily increase linearly as more of the video is observed. For TSA, similar ρ at 10–50% does not mean unchanged predictions: individual predicted scores change by 1.72 points on average.

(to be continued)

Q3 · Domain-dependent gains from density weighting

Response:

A possible explanation is the difference in action structure. Figure skating involves longer, multi-element routines, where similar prefixes may lead to diverse continuations. Conditional modelling may therefore provide information beyond prefix similarity. In FineDiving, candidates share the same dive number and have a more constrained action structure, so prefix similarity may already provide a strong compatibility cue, leaving less room for additional gains.

Reference construction using training data. Please see lines 247–254 of the paper for the definition and motivation of the reference correction. For FineFS, the reference pool contains training video prefixes. During training, at a fixed observation ratio x, let C be an ending from one training video, and let A and B be prefixes from two other training videos. If C appears in the candidate pools retrieved for A and B, these prefixes belong to C’s reference pool. The conditional densities of C under A and B have already been computed. We reuse their log-densities to calculate C’s reference value. If A and B are its only references, this is:

bC(x)=[log⁡p(C∣A,x)+log⁡p(C∣B,x)]/2.b_C(x) = [\log p(C \mid A, x) + \log p(C \mid B, x)] / 2.

In practice, we average over all available training prefixes whose candidate pools contain C. Using these existing pairs avoids extra density evaluations for reference correction.

Prediction for a test video. Let D be the observed prefix of a new test video at the same ratio x. If C is retrieved as a candidate ending for D, we compute its conditional log-density under D and subtract the stored training reference value:

sC(D,x)=log⁡p(C∣D,x)−bC(x).s_C(D, x) = \log p(C \mid D, x) - b_C(x).

The corrected values are converted into candidate weights using Eq. (9), with the temperature fixed before testing. Thus, C is still evaluated under the actual test prefix D; A and B only provide its reference value. The reference pool and correction parameters are computed using training data only. Test videos do not contribute to either.

The correction aims to reduce the preference for endings that consistently receive high density across the selected training prefixes. On FineFS, mean ρ rounds to 0.800 both with and without this correction, so reference correction does not explain the reported +0.09 gap.

Q4 · Uncertainty, error, and risk-coverage trade-off

Response:

For each test video, we use flow-based weights to combine candidate-completion scores into one final prediction ŷ. On FineDiving, the Spearman correlation between completion-score dispersion σ and absolute prediction error |ŷ − y| averages 0.30 across scoring models and observation ratios.

For risk–coverage analysis, we compare two ways of retaining final predictions across test videos: keeping those with the lowest σ, or randomly selecting the same amount of test videos. This does not change the candidate completions or their flow-based weights. We vary the number retained and measure their mean absolute error. On FineDiving, selecting by σ gives a lower area under the risk–coverage curve in 29 of 30 settings (six scorers × five observation ratios).

Q5 · Stability across truncation offsets and seeds

Response:

Truncation offsets. On FineFS/TSMP, we tested ±2 percentage points around 10%, 30%, and 50% observation, using the same scorer for both methods. A²QA outperforms Trunc at all nine cutoffs, with gains of 0.140–0.318 in ρ. Within each group of three cutoffs, the variation in ρ (maximum minus minimum) is:

Observation cutoffsA²QA variation in ρTrunc variation in ρ
8%, 10%, 12%0.0070.171
28%, 30%, 32%0.0570.035
48%, 50%, 52%0.0150.044

Training seeds. On FineDiving, three independent flow-training seeds with fixed scorers outperform Trunc in all 54 early-observation cases (six scorers × three ratios at 10/30/50% × three seeds). Across six scorers and five observation ratios, the mean sample standard deviation of ρ across seeds is 0.0022.

We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.

Q3: In the Updated Manuscript PDF, please see lines 257–264 of the paper for the definition and motivation of the reference correction, instead of lines 247–254 of the original PDF.

We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.

Reviewer_sfGeOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper studies anytime action quality assessment (AQA), aiming to predict the final performance score from a partially observed video while accounting for uncertainty in its unseen future. The authors propose A$^2$QA, a model-agnostic framework that retrieves real future segments from training videos with compatible declared content and appends them to the partial video in feature space. A conditional flow model estimates the conditional probability density of each candidate completion, and a frozen scoring model evaluates the completed sequences. The resulting scores are aggregated using likelihood-ratio weights to predict the final performance score. Experiments on four benchmarks across 13 model–dataset combinations demonstrate improvements in Spearman correlation over directly scoring truncated videos.

Critical strengths
  1. The proposed framework is model-agnostic, allowing existing pretrained AQA scoring models to be reused without retraining or modification.

  2. The experiments are relatively extensive, with consistent improvements over directly scoring truncated videos at early observation stages.

Critical weaknesses
  1. The paper does not adequately justify why predicting the final performance score before an action concludes is necessary. Although the authors motivate anytime AQA through prospective feedback, early final-score predictions do not directly provide actionable guidance for improving ongoing performance. Real-time assessment of observed actions may be more relevant to this goal, without requiring predictions of uncertain future actions.

  2. The proposed framework involves multiple computational steps, including retrieving candidate endings, estimating their conditional densities using a flow model, and evaluating the completed sequences with a pretrained AQA model. This complex inference pipeline can incur substantial latency and computational costs.

Questions for authors
  1. Could the authors clarify the practical motivation for predicting the final performance score from partially observed videos? In particular, what applications would benefit from early final-score prediction rather than assessing the observed actions or waiting until the performance is complete?

  2. Could the authors provide an analysis of the inference latency and computational costs of A²QA?

4 comments

We thank the reviewer for the positive assessment of our model-agnostic design, the reuse of pretrained scorers, and the extensive experiments showing consistent improvements at early observation stages. We address each question below.

Q1 · Applications that benefit from early score prediction

Response:

Early final-score prediction estimates whether an ongoing performance is likely to reach a target final score. For example, a sudden drop in the predicted final score could indicate a potential performance issue, prompting a coach to review recent actions and provide feedback before the routine ends. Live sports coverage could also use it to provide a running forecast of the final score.

Assessing observed actions describes performance so far. Our forecast estimates the overall outcome, including the unseen remainder. It complements action-level feedback; identifying specific corrections still requires a detailed assessment.

Other reviewers also recognize the value of this work: i7s1 states that “the problem is worth studying,” 93x5 notes its practical usefulness, and J9w6 highlights the reuse of pretrained scorers and early-prediction gains.

Q2 · Latency and cost analysis

Response:

Candidate completions are scored in parallel. On one RTX 3090, their scoring latency is at most 1.55× that of the original AQA model on truncated videos across all 13 model–dataset pairs (Table R2). Evaluating multiple completions therefore does not require sequential scorer runs.

For CoRe and Uni-FineParser, scoring uses cached features; feature extraction is included in Encode.

Density estimation takes 26.1–38.3 ms on FineDiving, JIGSAWS and LapEx, and 226 ms on FineFS.

The Full column ranges from 38.5 to 1,431.5 ms, with 11 of 13 model–dataset pairs below one second.

These timings suggest that A²QA can provide timely, periodic score updates for coaching feedback and live sports coverage.

Table R2. Inference latency (ms) on one NVIDIA RTX 3090. Full = Encode + Density + Scorer.

DatasetScorerTrunc.EncodeDensityScorer (parallel)Full
FineFSTSMP34.550222634.3762.3
GDLT2.95102262.9738.9
B2S31.451122648.6785.6
FineDivingTSA24.76.628.719.254.5
FineCausal1,38017.426.11,3881,431.5
HP-MCoRe76.36.628.776.3111.6
CoRe8.012.630.51.444.5
RICA²25.213.228.725.267.1
Uni-FineParser16.921.438.31.461.1
JIGSAWSCoRe43.66.130.71.738.5
RICA²1,08811.630.71,0801,122.3
LapExMoAKE12.55.732.912.551.1
LIMSSR53.75.732.953.792.3

We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.

Thank you for the detailed responses and additional experimental results. The reported inference latency measurements address my concern regarding inference efficiency. I have therefore increased my score from 1 to 2.

Previously I thought the goal of early performance feedback can be addressed through real-time assessment of observed actions. I now see that A2QA offers a different perspective by forecasting the final score based on possible future actions. This can inspire some more future works in this direction. Also, I would appreciate it if the authors could further clarify the potential advantages of considering future performance over relying solely on observed actions, particularly in practical applications. If this concern could be fully addressed, I would like to to further improve the score.

We would like to thank the reviewer for the constructive follow-up.

The practical benefit is assessing whether an ongoing performance is on track to reach a final-score target. A²QA considers plausible continuations to estimate the final result, including the unfinished portion.

In figure skating, two routines may receive similar assessments for their observed actions, but one may still have several difficult jumps ahead. Their final-score outlooks can differ because the remaining elements and their possible execution quality differ. In training, this forecast could help a coach prioritize attention to routines predicted to miss their targets.

In simulated suturing practice, a trainee may perform the initial steps well while more demanding steps remain. Forecasting estimates the overall quality of the finished exercise and could help an instructor decide which ongoing attempts need closer attention. Local assessment can then help identify specific execution issues.

During a diving broadcast, a diver may perform the take-off and flight well, but the entry is still unseen. By considering possible entry outcomes, a final-score forecast could help commentators discuss whether the ongoing dive is on track to exceed a leading score, beyond judging only the phases already observed.

We will clarify these potential applications in the revised paper. Thank you again for the helpful discussion.

Thank you for further clarifying the motivation and the value of the problem. Now I do not have major concerns, so I am raising the score again.

Reviewer_93x5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes A2QA, a framework for predicting the final quality score of an action from a partially observed video. It retrieves compatible endings from training videos, concatenates them with the observed prefix in feature space, and evaluates the resulting completions using a frozen action-quality scorer. A conditional flow model determines how much each completion contributes to the final score prediction, while the weighted variation among completion scores is used to estimate uncertainty. The included experiments cover four datasets spanning different domains, including sports and surgical skill assessment.

The main contribution is a modular combination of retrieval-based completion, conditional density estimation, and existing scoring models for early performance assessment. This design preserves the original scorer and recovers its prediction at full observation. Evaluation across 13 model-dataset combinations demonstrates consistent improvements over directly scoring truncated videos at early observation stages. Overall, this work presents a promising application oriented contribution.

Critical strengths

[S1] The framework enables existing pretrained action-quality models to predict final scores from partially observed videos without modifying or retraining those models.

The authors propose a novel approach that demonstrates good results using existing pretrained models. By separating future completion from quality scoring, A2QA adapts existing pretrained video models to perform partial observations without modifying or retraining the underlying scorer. The framework also recovers the original scorer’s prediction at full observation. It ensures that adding early prediction capability does not alter the pretrained model’s output when the complete video becomes available, preserving its existing full-video performance.

[S2] The evaluation results demonstrate consistent improvements across diverse scoring models and application domains.

Experiments cover 13 model-dataset combinations across different domains, including four sports and surgical assessment datasets, with improvements over direct truncated-video scoring in every comparison at 10%, 30%, and 50% observation. Ablations against uniform weighting, prefix similarity, and uncorrected density weighting provide additional supporting evidence for the contribution of the proposed aggregation mechanism.

Critical weaknesses

I lean toward acceptance because the framework offers a potentially useful and practical extension of existing scorers and demonstrates broad early-observation gains. My main reservations concern the specification of prediction-time information and the breadth of baseline comparisons, which should be addressed during rebuttal.

[W1] The paper does not clearly establish what information is available when an early prediction is made.

Within the proposed framework: (i) retrieval depends on declared future actions, and (ii) the observation ratio depends on the performance’s total duration. The authors should clarify how this information is obtained for each dataset and whether it is available during an ongoing performance. Without these details, it is difficult to determine how much improvement comes from the proposed framework and observed video, versus advance or other additional metadata that may not always be available. This could potentially limit the applicability of the proposed framework.

[W2] The results provided do not directly compare to equivalent models trained specifically for early score prediction.

The existing experiments do not show whether A2QA outperforms models specifically trained to predict final scores directly from partially observed videos. The main baseline for comparison applies existing pretrained full-video scorers directly to truncated inputs. A2QA supplements the scorer with a trained flow model, score-supervised temperature selection, and retrieval based on action-category and declared future-content metadata. The paper does not clearly establish whether the baselines receive the same (or otherwise equivalent) metadata. Additional comparisons with a regressor trained on partial observations or a nearest-neighbor score predictor, using the same features and metadata, would better establish the benefit of the proposed completion and weighting mechanism by comparing to other similar models as opposed to existing pretrained video models.

Questions for authors

[Q1] What action-category, declared-content, and total-duration information is available at prediction time for each dataset, and do all baselines receive equivalent information?

[Q2] How does A2QA compare with a model trained to predict final scores directly from partial videos using the same features and metadata, and with a baseline using only declared-content metadata?

[Q3] Does the proposed completion-score dispersion reliably predict actual scoring error, and can this be demonstrated through uncertainty-error analysis or risk-coverage curves?

3 comments

We thank the reviewer for the positive assessment of the paper’s clarity, the practical value of our framework, its model-agnostic design, and the consistent improvements across models and domains. We address each question below.

Q1 · Prediction-time information and baseline

Response:

Our experiments use the following information:

  • FineDiving: The annotated dive number.
  • FineFS: Short/free program type and remaining element classes and base values.
  • JIGSAWS: The surgical task identity.
  • LapEx: The annotated action verb–target pair at the assessed instant.

Duration. For an ongoing performance, we use each action category’s average or standard duration.

Baseline information. Each baseline uses its original input settings.

Q2 · Partial-video and declared-content-only baselines

Response:

Table R1 includes fine-tuning (FT), intermediate supervision (IS), and a zero-video group-mean predictor (Prior). Prior uses the mean training score of the same category/declaration group. At 10%, 30%, and 50% observation, A²QA achieves higher ρ than FT in 38/39 settings, IS in 39/39, and Prior in 35/39.

(to be continued)

Table R1. ρ and R-ℓ₂ × 100 at 10%, 30%, and 50% observation.

Spearman’s ρ (↑)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
SkatingTSMP0.3950.5130.5740.7590.3950.6610.6100.5750.5800.7590.6100.7770.5940.6170.6720.7590.5940.795
GDLT0.5790.6340.5740.7590.5790.7960.5890.6930.5800.7590.5890.8060.7230.7710.6720.7590.7230.805
B2S0.7680.7760.5740.7590.7680.8050.8270.8280.5800.7590.8270.8420.8480.8390.6720.7590.8480.857
DivingTSA0.5530.5860.4010.5880.5530.6120.5570.5970.5100.5880.5570.6380.5730.6150.5560.5880.5730.635
FineCausal0.5440.5570.4010.5880.5440.6140.5570.6020.5100.5880.5570.6400.5550.6130.5560.5880.5550.646
HP-MCoRe0.5790.5660.4010.5880.5790.6090.5860.5970.5100.5880.5860.6300.5940.6150.5560.5880.5940.635
CoRe0.5490.5750.4010.5880.5490.6000.5730.6060.5100.5880.5730.6110.5820.6130.5560.5880.5820.634
RICA²0.5870.5900.4010.5880.5870.6240.5960.6030.5100.5880.5960.6480.5860.5950.5560.5880.5860.651
Uni-FineParser0.5680.5900.4010.5880.5680.6040.5910.6290.5100.5880.5910.6250.6140.6420.5560.5880.6140.645
SurgeryCoRe0.3960.4230.218−0.2140.5190.6270.6580.6790.259−0.2140.7020.6960.6830.6700.359−0.2140.6970.720
RICA²0.4140.3770.218−0.2140.3770.5960.5640.5510.259−0.2140.5560.5750.6390.6240.359−0.2140.6160.676
LapExMoAKE0.1710.1200.0070.2240.1710.1990.1900.0930.0480.2240.1900.2440.2400.1270.1460.2240.2400.274
LIMSSR0.0560.0810.0070.2240.0560.1170.1030.1360.0480.2240.1030.1790.1370.1940.1460.2240.1370.232

R-ℓ₂ × 100 (↓)

DomainScoring model10% Trunc.10% (a) FT10% (a) IS10% (b) Prior10% (c) Recal.10% Ours30% Trunc.30% (a) FT30% (a) IS30% (b) Prior30% (c) Recal.30% Ours50% Trunc.50% (a) FT50% (a) IS50% (b) Prior50% (c) Recal.50% Ours
DivingTSA4.731.362.971.341.381.304.491.342.041.341.381.274.111.311.721.341.361.26
FineCausal2.131.352.971.341.351.232.001.282.041.341.351.212.051.251.721.341.341.18
HP-MCoRe3.031.452.971.341.341.302.551.372.041.341.321.251.961.321.721.341.301.22
CoRe1.851.642.971.341.391.261.771.552.041.341.351.231.811.601.721.341.331.19
RICA²1.531.312.971.341.291.211.571.322.041.341.271.161.651.381.721.341.291.15
Uni-FineParser1.951.342.971.341.361.311.861.292.041.341.331.261.801.291.721.341.291.21

FT: fine-tuning on truncated inputs; IS: intermediate supervision; Prior: zero-video group mean; Recal.: linear recalibration. Bold indicates the best value for each model and observation ratio.

(to be continued)

Q3 · Uncertainty and actual prediction error

Response:

For each test video, we use flow-based weights to combine candidate-completion scores into one final prediction ŷ. On FineDiving, the Spearman correlation between completion-score dispersion σ and absolute prediction error |ŷ − y| averages 0.30 across scoring models and observation ratios.

For risk–coverage analysis, we compare two ways of retaining final predictions across test videos: keeping those with the lowest σ, or randomly selecting the same amount of test videos. This does not change the candidate completions or their flow-based weights. We vary the number retained and measure their mean absolute error. On FineDiving, selecting by σ gives a lower area under the risk–coverage curve in 29 of 30 settings (six scorers × five observation ratios).

We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript PDF

We have uploaded the updated manuscript PDF. Updated contents have been highlighted in blue. We sincerely thank the reviewers for their time and constructive feedback. We would also be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.