- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Summary of contributions:
- The authors propose an evaluation framework for deployment-facing action quality assessment systems, focusing on repeated measurements, personalization, human-assisted review, physical measurement validity, and refusal.
- Their “evidence contract” associates each system capability or reliability claim with an appropriate evaluation check, including user-level aggregation, separation of calibration and testing, blind reference labels, physical measurement agreement, and error analysis across refusal coverage.
- The framework is illustrated through a retrospective audit of an eight-month mobile rehabilitation project. The reported audits expose substantial differences in apparent performance: calibration accuracy is 91.8% under same-user fitting and scoring versus 55.7% under held-out-user evaluation; online agreement falls from 93.7% with pooled repetitions to 74.8% with equal user weighting; and agreement on 23 matched videos falls from 91.3% against model-visible judgments to 39.1% against blind consensus.
Critical strengths
Strengths:
- Practical relevance: Reliable assessment of rehabilitation exercises is an important application with substantial potential utility. The paper addresses consequential evaluation issues that arise when systems personalize their outputs, involve human reviewers, and operate under imperfect observation conditions. Examining these issues in an operational project provides useful motivation for more rigorous deployment evaluation.
Critical weaknesses
Weaknesses:
- Lack of clarity in presentation: The paper requires substantial effort to understand. The Abstract and Discussion introduce numerous metrics and broad methodological claims before clearly establishing the system’s inputs, outputs, reference labels, and evaluation units. The meaning of an “evidence contract,” its relationship to existing evaluation practices, and the specific contribution it enables should be explained more directly. Each quantitative result should identify what is being predicted, how correctness is determined, and whether aggregation occurs over repetitions, videos, rules, or users. The current transitions among these settings make the results difficult to interpret.
- Unclear contribution and advance over prior work: The manuscript does not sufficiently distinguish the development of the assessment system, the audit of its internal evaluation practices, and the methodological novelty claimed for the evidence contract. Many constituent checks—held-out-user evaluation, temporal separation of adaptation and testing, blind labeling, measurement agreement, and selective-risk analysis—are established practices. The authors should more clearly describe existing AQA evaluation protocols and explain what their proposed integration adds. The matched retrospective comparisons are informative, but broader claims would be strengthened by demonstrating the framework’s value in another system or an independent prospective evaluation, using appropriate expert or physical references.
- Role of automation and agentic alternatives: The assessed system relies substantially on hand-designed measurements, rules, thresholds, and human review. It would be useful to clarify which parts of the assessment and audit workflow could be automated, including through agentic systems, and what tradeoffs such automation introduces. The limited VLM pilot provides an initial comparison, but leaves the broader role of automated workflows underexplored. Any such comparison should retain independent reference labels or measurements, since automation alone would not resolve the evaluation dependencies identified by the paper.
Questions for authors
Please refer to my weaknesses review above.
Dear Reviewer ZNH5,
Thank you for reading our manuscript carefully. We appreciate your recognition that a real rehabilitation project can shed light on evaluation challenges involving personalization, human review, and incomplete observations. We have made targeted changes addressing your comments on presentation, contribution, and automation for the revised manuscript.
Thank you again for your review and constructive feedback.