A${}^2$QA: Anytime Action Quality Assessment via Flow-Based Score Forecasting

Action Quality Assessment (AQA) typically predicts a final performance score only after observing the complete video. However, in sports and skill training, predicting performance outcomes before an action concludes can enable earlier and more prospective feedback. We study the anytime AQA task in this paper, where the goal is to predict the final score from a partially observed performance while accounting for uncertainty in its unseen future. We propose A2QA, a model-agnostic framework that augments a pretrained AQA model with plausible future completions. Given an observed prefix, A2QA retrieves real future segments from training videos with compatible declared content and appends them to the prefix in feature space. To model uncertainty over possible futures, we introduce a conditional flow model that estimates the likelihood of each retrieved completion conditioned on the observed prefix. A frozen scoring model then evaluates each completed sequence, and their likelihood-weighted scores are aggregated to predict the final performance score. The framework requires neither retraining nor modification of the underlying scoring model and naturally reduces to the original model when the full performance is observed. Experiments on four datasets spanning sports and surgical skill assessment (FineFS, FineDiving, JIGSAWS, and LapEx) show consistent improvements across all 13 model. For example, in TSMP on FineFS datasets, Spearman correlation improves from 0.395 to 0.661 at only 10% observation. These results demonstrate that explicitly modeling plausible future completions substantially improves final-score prediction from partial observations while preserving compatibility with existing AQA models.