- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
A²QA predicts the final action quality score from a partial video. It appends compatible endings from training videos to the observed prefix in feature space, scores each completion with a frozen AQA model, and averages the scores using weights derived from conditional flow densities. With the full video available, it returns the original model’s prediction. Across four datasets and 13 model–dataset pairs, A²QA outperforms direct scoring of truncated videos at 10%, 30%, and 50% observation. FineDiving results also show lower numerical scoring errors.
Critical strengths
-
A²QA reuses a pretrained full-video scorer while learning how to weight alternative real endings. This avoids requiring a new scoring architecture for every backbone and avoids synthesizing future video. Returning the original prediction at full observation also gives the framework a clear relationship to conventional AQA.
-
Table 1 reports improvements at 10%, 30%, and 50% observation for all 13 model–dataset pairs. For example, TSMP on FineFS improves from 0.395 to 0.661 in Spearman correlation at 10% observation. FineDiving results include numerical score errors as well as ranking correlation. Table 3 further shows that learned density weighting improves over uniform and prefix-similarity weighting on skating and diving. These results support the effectiveness of the approach under the tested conditions.
Critical weaknesses
-
The main baseline scores truncated videos with a model trained on complete videos. This comparison shows the benefit of adapting a frozen scorer, but does not establish an advantage over models trained for early prediction. Despite discussing prior partial-observation scoring work, the paper includes no such baseline. A comparison with a model trained on prefixes using final-score supervision should match information access and report training and inference costs.
-
The method depends on compatible categories, declared content, and training-library coverage. Joining segments from different performances may produce implausible transitions, whose effect on the scorer is not tested. Low completion-score disagreement may also hide large errors when plausible futures are missing; Figure 6 does not establish calibration or reliable prediction withholding. Inference evaluates up to 30 candidates through density integration and scoring, but end-to-end latency is not reported.
Questions for authors
-
How does A²QA compare with a predictor trained on randomly truncated videos using final-score supervision, and with applicable partial-observation AQA methods discussed in Section 2? Please match the observed inputs and access to action/declaration information, and report differences in training requirements and inference cost. This would clarify the trade-off between reusing a frozen scorer and training specifically for early prediction.
-
How does performance change as the number or diversity of available candidate endings decreases, or when the action declaration is unavailable or incorrect? What happens if there is no compatible candidate? Can you provide evidence that incompatible transitions between the observed prefix and a retrieved ending do not systematically mislead the frozen scorer?
4 comments
Table R1. ρ and R-ℓ₂ × 100 at 10%, 30%, and 50% observation.
Spearman’s ρ (↑)
| Domain | Scoring model | 10% Trunc. | 10% (a) FT | 10% (a) IS | 10% (b) Prior | 10% (c) Recal. | 10% Ours | 30% Trunc. | 30% (a) FT | 30% (a) IS | 30% (b) Prior | 30% (c) Recal. | 30% Ours | 50% Trunc. | 50% (a) FT | 50% (a) IS | 50% (b) Prior | 50% (c) Recal. | 50% Ours |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Skating | TSMP | 0.395 | 0.513 | 0.574 | 0.759 | 0.395 | 0.661 | 0.610 | 0.575 | 0.580 | 0.759 | 0.610 | 0.777 | 0.594 | 0.617 | 0.672 | 0.759 | 0.594 | 0.795 |
| GDLT | 0.579 | 0.634 | 0.574 | 0.759 | 0.579 | 0.796 | 0.589 | 0.693 | 0.580 | 0.759 | 0.589 | 0.806 | 0.723 | 0.771 | 0.672 | 0.759 | 0.723 | 0.805 | |
| B2S | 0.768 | 0.776 | 0.574 | 0.759 | 0.768 | 0.805 | 0.827 | 0.828 | 0.580 | 0.759 | 0.827 | 0.842 | 0.848 | 0.839 | 0.672 | 0.759 | 0.848 | 0.857 | |
| Diving | TSA | 0.553 | 0.586 | 0.401 | 0.588 | 0.553 | 0.612 | 0.557 | 0.597 | 0.510 | 0.588 | 0.557 | 0.638 | 0.573 | 0.615 | 0.556 | 0.588 | 0.573 | 0.635 |
| FineCausal | 0.544 | 0.557 | 0.401 | 0.588 | 0.544 | 0.614 | 0.557 | 0.602 | 0.510 | 0.588 | 0.557 | 0.640 | 0.555 | 0.613 | 0.556 | 0.588 | 0.555 | 0.646 | |
| HP-MCoRe | 0.579 | 0.566 | 0.401 | 0.588 | 0.579 | 0.609 | 0.586 | 0.597 | 0.510 | 0.588 | 0.586 | 0.630 | 0.594 | 0.615 | 0.556 | 0.588 | 0.594 | 0.635 | |
| CoRe | 0.549 | 0.575 | 0.401 | 0.588 | 0.549 | 0.600 | 0.573 | 0.606 | 0.510 | 0.588 | 0.573 | 0.611 | 0.582 | 0.613 | 0.556 | 0.588 | 0.582 | 0.634 | |
| RICA² | 0.587 | 0.590 | 0.401 | 0.588 | 0.587 | 0.624 | 0.596 | 0.603 | 0.510 | 0.588 | 0.596 | 0.648 | 0.586 | 0.595 | 0.556 | 0.588 | 0.586 | 0.651 | |
| Uni-FineParser | 0.568 | 0.590 | 0.401 | 0.588 | 0.568 | 0.604 | 0.591 | 0.629 | 0.510 | 0.588 | 0.591 | 0.625 | 0.614 | 0.642 | 0.556 | 0.588 | 0.614 | 0.645 | |
| Surgery | CoRe | 0.396 | 0.423 | 0.218 | −0.214 | 0.519 | 0.627 | 0.658 | 0.679 | 0.259 | −0.214 | 0.702 | 0.696 | 0.683 | 0.670 | 0.359 | −0.214 | 0.697 | 0.720 |
| RICA² | 0.414 | 0.377 | 0.218 | −0.214 | 0.377 | 0.596 | 0.564 | 0.551 | 0.259 | −0.214 | 0.556 | 0.575 | 0.639 | 0.624 | 0.359 | −0.214 | 0.616 | 0.676 | |
| LapEx | MoAKE | 0.171 | 0.120 | 0.007 | 0.224 | 0.171 | 0.199 | 0.190 | 0.093 | 0.048 | 0.224 | 0.190 | 0.244 | 0.240 | 0.127 | 0.146 | 0.224 | 0.240 | 0.274 |
| LIMSSR | 0.056 | 0.081 | 0.007 | 0.224 | 0.056 | 0.117 | 0.103 | 0.136 | 0.048 | 0.224 | 0.103 | 0.179 | 0.137 | 0.194 | 0.146 | 0.224 | 0.137 | 0.232 |
R-ℓ₂ × 100 (↓)
| Domain | Scoring model | 10% Trunc. | 10% (a) FT | 10% (a) IS | 10% (b) Prior | 10% (c) Recal. | 10% Ours | 30% Trunc. | 30% (a) FT | 30% (a) IS | 30% (b) Prior | 30% (c) Recal. | 30% Ours | 50% Trunc. | 50% (a) FT | 50% (a) IS | 50% (b) Prior | 50% (c) Recal. | 50% Ours |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Diving | TSA | 4.73 | 1.36 | 2.97 | 1.34 | 1.38 | 1.30 | 4.49 | 1.34 | 2.04 | 1.34 | 1.38 | 1.27 | 4.11 | 1.31 | 1.72 | 1.34 | 1.36 | 1.26 |
| FineCausal | 2.13 | 1.35 | 2.97 | 1.34 | 1.35 | 1.23 | 2.00 | 1.28 | 2.04 | 1.34 | 1.35 | 1.21 | 2.05 | 1.25 | 1.72 | 1.34 | 1.34 | 1.18 | |
| HP-MCoRe | 3.03 | 1.45 | 2.97 | 1.34 | 1.34 | 1.30 | 2.55 | 1.37 | 2.04 | 1.34 | 1.32 | 1.25 | 1.96 | 1.32 | 1.72 | 1.34 | 1.30 | 1.22 | |
| CoRe | 1.85 | 1.64 | 2.97 | 1.34 | 1.39 | 1.26 | 1.77 | 1.55 | 2.04 | 1.34 | 1.35 | 1.23 | 1.81 | 1.60 | 1.72 | 1.34 | 1.33 | 1.19 | |
| RICA² | 1.53 | 1.31 | 2.97 | 1.34 | 1.29 | 1.21 | 1.57 | 1.32 | 2.04 | 1.34 | 1.27 | 1.16 | 1.65 | 1.38 | 1.72 | 1.34 | 1.29 | 1.15 | |
| Uni-FineParser | 1.95 | 1.34 | 2.97 | 1.34 | 1.36 | 1.31 | 1.86 | 1.29 | 2.04 | 1.34 | 1.33 | 1.26 | 1.80 | 1.29 | 1.72 | 1.34 | 1.29 | 1.21 |
FT: fine-tuning on truncated inputs; IS: intermediate supervision; Prior: zero-video group mean; Recal.: linear recalibration. Bold indicates the best value for each model and observation ratio.
(to be continued)
Table R2. Inference latency (ms) on one NVIDIA RTX 3090. Full = Encode + Density + Scorer.
| Dataset | Scorer | Trunc. | Encode | Density | Scorer (parallel) | Full |
|---|---|---|---|---|---|---|
| FineFS | TSMP | 34.5 | 502 | 226 | 34.3 | 762.3 |
| GDLT | 2.9 | 510 | 226 | 2.9 | 738.9 | |
| B2S | 31.4 | 511 | 226 | 48.6 | 785.6 | |
| FineDiving | TSA | 24.7 | 6.6 | 28.7 | 19.2 | 54.5 |
| FineCausal | 1,380 | 17.4 | 26.1 | 1,388 | 1,431.5 | |
| HP-MCoRe | 76.3 | 6.6 | 28.7 | 76.3 | 111.6 | |
| CoRe | 8.0 | 12.6 | 30.5 | 1.4 | 44.5 | |
| RICA² | 25.2 | 13.2 | 28.7 | 25.2 | 67.1 | |
| Uni-FineParser | 16.9 | 21.4 | 38.3 | 1.4 | 61.1 | |
| JIGSAWS | CoRe | 43.6 | 6.1 | 30.7 | 1.7 | 38.5 |
| RICA² | 1,088 | 11.6 | 30.7 | 1,080 | 1,122.3 | |
| LapEx | MoAKE | 12.5 | 5.7 | 32.9 | 12.5 | 51.1 |
| LIMSSR | 53.7 | 5.7 | 32.9 | 53.7 | 92.3 |
Q2 · Candidate availability, declarations, and transition robustness
Response:
Candidate number and diversity. On FineFS/TSMP at 10% observation, reducing the candidate count from 30 to 3 or 1 lowers mean ρ from 0.661 to 0.527 or 0.392. With 15 candidates, a less diverse set of similar endings gives ρ of 0.606, versus 0.641 for random selection.
Missing or incorrect declarations. To test missing declarations, we retrieve candidates from all training videos using prefix similarity. For TSMP on FineFS at 10% observation, ρ decreases from 0.661 with declarations to 0.513 without them. Giving TSMP the wrong label lowers ρ at 10% observation from 0.661 to −0.268.
No compatible candidate. If the pool is empty, the method cannot produce a completion-based score; handling this case remains future work.
Transitions at the join. On FineFS/TSMP at 10% observation, smoothing one or four feature rows at the join gives ρ of 0.656 or 0.643, versus 0.661 before smoothing and 0.395 for Trunc. The ranking advantage remains under these local changes.
We thank the reviewer again for the careful reading and hope these answers and the added analyses address the concerns; we are happy to provide any further clarification.
The added comparisons with fine-tuning and intermediate supervision strengthen the evidence for A²QA, while the latency measurements and robustness experiments address several of my concerns. However, equivalent access to action and declaration information across baselines remains unclear, and the method still depends strongly on candidate coverage and correct declarations. The transition and uncertainty analyses provide useful but limited evidence of reliability. I therefore maintain my rating of 3 (Weak acceptance).
We thank the reviewer for the positive assessment of the paper’s clarity, our reuse of pretrained scorers, and the consistent improvements across models and datasets. We address each question below.
Q1 · Comparison with trained partial-observation methods
Response:
Table R1 shows higher ρ for A²QA than fine-tuning (FT) in 38/39 settings and intermediate supervision (IS) in 39/39 settings at 10–50% observation.
Both training and inference for A²QA, FT, and IS can be performed on a single RTX 3090.
FT/IS train a predictor on partial videos. A²QA keeps the scorer frozen and trains a flow model. Its inference adds retrieval, density estimation, and scoring of up to 30 completions. Table R2 reports the encoding, density-estimation, and scoring times.
(to be continued)