acceptodds
Under review as a conference paper at ICLR 2027

Measure, Calibrate, Refuse and Verify: An Evidence Contract for Mobile AQA

Abstract

Deployment-facing action quality assessment combines repeated observations, user adaptation, human review, physical measurements, and abstention, yet these components are often evaluated as if their evidence were independent. We formulate an *evidence contract*: every adaptive or human-assisted degree of freedom requires a matching source of evaluation independence. We instantiate the contract in a longitudinal audit of an eight-month mobile rehabilitation system and obtain four paired reversals. Same-user threshold fitting raised accuracy from to , whereas nested leave-one-user-out calibration achieved . Pooled repetition agreement of became when users received equal weight. On the same 23 videos and frozen predictions, model agreement changed from under model-visible review to against blind consensus; 51 videos were retrospectively labeled, of which 38 entered frozen-rule scoring after exclusions, yielding error-detection accuracies of and for two exercises. Finally, within-user correlation of coexisted with centimeter limits of agreement , while refusal retained user-level risk at coverage. The resulting evaluation stack—user-level aggregation, nested adaptation, blind-first labeling, measurement agreement, and coverage–risk analysis—turns apparently reliable outputs into testable stage-wise claims. It applies beyond rehabilitation wherever systems personalize, expose machine suggestions to reviewers, or withhold predictions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
action quality assessment, rehabilitation, calibration, personalization, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZNH5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary of contributions:

  • The authors propose an evaluation framework for deployment-facing action quality assessment systems, focusing on repeated measurements, personalization, human-assisted review, physical measurement validity, and refusal.
  • Their “evidence contract” associates each system capability or reliability claim with an appropriate evaluation check, including user-level aggregation, separation of calibration and testing, blind reference labels, physical measurement agreement, and error analysis across refusal coverage.
  • The framework is illustrated through a retrospective audit of an eight-month mobile rehabilitation project. The reported audits expose substantial differences in apparent performance: calibration accuracy is 91.8% under same-user fitting and scoring versus 55.7% under held-out-user evaluation; online agreement falls from 93.7% with pooled repetitions to 74.8% with equal user weighting; and agreement on 23 matched videos falls from 91.3% against model-visible judgments to 39.1% against blind consensus.
Critical strengths

Strengths:

  • Practical relevance: Reliable assessment of rehabilitation exercises is an important application with substantial potential utility. The paper addresses consequential evaluation issues that arise when systems personalize their outputs, involve human reviewers, and operate under imperfect observation conditions. Examining these issues in an operational project provides useful motivation for more rigorous deployment evaluation.
Critical weaknesses

Weaknesses:

  • Lack of clarity in presentation: The paper requires substantial effort to understand. The Abstract and Discussion introduce numerous metrics and broad methodological claims before clearly establishing the system’s inputs, outputs, reference labels, and evaluation units. The meaning of an “evidence contract,” its relationship to existing evaluation practices, and the specific contribution it enables should be explained more directly. Each quantitative result should identify what is being predicted, how correctness is determined, and whether aggregation occurs over repetitions, videos, rules, or users. The current transitions among these settings make the results difficult to interpret.
  • Unclear contribution and advance over prior work: The manuscript does not sufficiently distinguish the development of the assessment system, the audit of its internal evaluation practices, and the methodological novelty claimed for the evidence contract. Many constituent checks—held-out-user evaluation, temporal separation of adaptation and testing, blind labeling, measurement agreement, and selective-risk analysis—are established practices. The authors should more clearly describe existing AQA evaluation protocols and explain what their proposed integration adds. The matched retrospective comparisons are informative, but broader claims would be strengthened by demonstrating the framework’s value in another system or an independent prospective evaluation, using appropriate expert or physical references.
  • Role of automation and agentic alternatives: The assessed system relies substantially on hand-designed measurements, rules, thresholds, and human review. It would be useful to clarify which parts of the assessment and audit workflow could be automated, including through agentic systems, and what tradeoffs such automation introduces. The limited VLM pilot provides an initial comparison, but leaves the broader role of automated workflows underexplored. Any such comparison should retain independent reference labels or measurements, since automation alone would not resolve the evaluation dependencies identified by the paper.
Questions for authors

Please refer to my weaknesses review above.

1 comment

Dear Reviewer ZNH5,

Thank you for reading our manuscript carefully. We appreciate your recognition that a real rehabilitation project can shed light on evaluation challenges involving personalization, human review, and incomplete observations. We have made targeted changes addressing your comments on presentation, contribution, and automation for the revised manuscript.


  1. W1: Inputs, outputs, and units of evaluation. The revision will open with a bird-dog knee lift. From a single-camera clip, the system estimates knee-lift height, returns a verdict and a supporting frame, and abstains from issuing a verdict when there is insufficient visual evidence. Table 1 summarizes the references and units for each dataset, and the results will distinguish repetition-, video-, rule-, and user-level statistics.
  2. W2: What the integration adds and where the results apply. The individual evaluation methods are established. We apply them to the same pipeline to distinguish when the system needs to change from when the evaluation does. For example, with predictions and review labels held fixed, giving each user equal weight instead of pooling repetitions changes agreement from 0.937 to 0.748. The difference comes from aggregation, not from a change in the predictions. The revision will distinguish our evaluation of the reliability of each feedback output from the standard action quality assessment (AQA) task of predicting expert scores. It will also outline a prospective study with frozen rules, new users, and independent references, without extrapolating the retrospective findings to other systems.
  3. W3: Automation and its trade-offs. Pose extraction, threshold replay, and metric computation can be automated. In our pilot, the vision–language model and the rule disagreed on 24 of 61 videos; each was correct on some of those cases. The revision will explain how to test whether a model or agent can prioritize clips for review, comparing review workload and the risk of missed errors. Independent labels or grid readings would be recorded first, rather than using another model’s judgment as the reference.

Thank you again for your review and constructive feedback.

Reviewer_5zxSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes the concept of an "evidence contract" mobile action quality assessment. Basically, every part of the system built for a specific task (in this case, using a camera for monitoring physical rehabilitation activities) needs to have some kind of evaluation condition. For example, there needs to be held out data for adapting to new users, blind labels for expert review, or an objective reference for measuring physical activities. The work applies this concept to an eight month audit of a rehabilitation system, and identifies several trends as a result of the checks in place.

Critical strengths

The paper decomposes systems into different parts, providing some sort of visibility into the reasons behind system failures. Rather than just tuning model thresholds or parameters, the system can give insight into where in the pipeline the disagreements arise, allowing you to make more targeted fixes.

Critical weaknesses
  1. The paper is difficult to understand. For example, the abstract lists a bunch of changes in performance of various aspects of the system as a result of some change that was made, but it is unclear why those changes are made and what their significance is. Even after reading the evaluation, it is still very unclear. Another aspect that is confusing is that this system is supposed to incorporate human feedback at some point in the process, but it is unclear where and how it is incorporated (is it across all stages?); the term "audit" is used basically everywhere, making it seem like there is some human doing the auditing, but many of the "audits" seem to be automated? Basically, aside from the fact that there are "audits" placed at different parts of the pipeline, nothing else is clear to me.

  2. Novelty and claims seem limited and unsubstantiated. It seems most parts of the added "evidence contract" that supplements each part of the pipeline are borrowed from existing work, and there is no comparisons against this literature at all. For example, several works propose some sort of use case aligned validation with participant separation (Saeb et al.); DECIDE-AI does stage-specific evaluation with human factors, which is similar to this paper. Many of the results are descriptive and difficult to contextualize with respect to the experiments presented. Finally, this framework was only applied to a single application scenario (physical-rehabilitation), despite claims that it can generalize to other domains.

Questions for authors
  1. The only question that could really sway me is to elaborate and concisely state the core contribution and results. The paper in general needs to be overhauled in its presentation. Even if the presentation were to be improved, I think it'd be difficult elaborate on the core novelty. This work seems to be trying to propose something general, but also specifically applied to action quality assessment, and requires substantial reframing.

  2. The experimental results need to be more concisely stated. I assume they are supposed to be used to support the primary contribution of the proposed system, but currently, it's difficult to understand what the proposed "evidence contract" is supposed to provide to adopters of it.

  3. It may be helpful to see this framework applied to another application.

1 comment

Dear Reviewer 5zxS,

Thank you for your careful reading and concrete suggestions. We appreciate your point that evaluating the pipeline stage by stage can reveal where disagreements arise and guide targeted fixes, rather than simply prompting more threshold tuning. In response to your comments on clarity and the scope of the contribution, we have made changes to the abstract, methods, and discussion for the revised manuscript.


  1. W1 & Q2: How the system works and where people are involved. The initial verdict is automatic. A person can later confirm or correct it, and those reviewed labels can be used to fit candidate thresholds. In H1, by contrast, raters independently judged historical videos without seeing the system output. The evaluation metrics are computed automatically. We will use the same knee-lift example in the abstract and methods and identify the reference and unit of analysis for each dataset in Table 1.
  2. W2 & Q1–Q2: Core contribution and relation to prior work. The evidence contract specifies the reference and evaluation conditions needed to assess the reliability of each output, so developers can tell which part of the system needs attention. It does not introduce a new statistical test. For example, the 95% limits of agreement between estimated knee height and grid measurements are [−7.1, +4.2] cm, extending beyond the ±4 cm coaching tolerance. That points to checking the measurement first rather than simply refitting the decision threshold. Following Saeb et al., we match the data split to the intended use, distinguishing transfer to new users from adaptation for the same user on later repetitions. DECIDE-AI guides early live clinical evaluation; our study instead uses retrospective comparisons within one system to locate sources of disagreement. We will make this distinction explicit in the revision.
  3. Q3: Other applications. We agree that testing the framework in another setting would be valuable. The revision will limit the empirical conclusions to this rehabilitation setting and explain how to choose independent references and separate calibration from testing in another application.

Thank you again for your review and suggestions.

Reviewer_9Bp9OpenReview ↗
Rating
2(−1)
Confidence
2(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits an eight-month mobile rehabilitation system and asks what evidence is needed for its measurements, calibration, human review and refusal decisions. Several encouraging development results change sharply when the authors reweight by user, restrict calibration access, relabel videos blind, or compare physical estimates with a reference. These checks are organized into an “evidence contract” linking each system component to the evaluation needed to support its claim. The study is retrospective and does not claim prospective clinical validation.

Critical strengths
  1. The most useful part is that the authors revisit their own favorable results while keeping predictions fixed where possible. Online agreement drops from 0.937 to 0.748 after balancing users; on the same 23 videos it changes from 0.913 with model-visible review to 0.391 against blind consensus. Seeing these failures together makes the limits of a single end-to-end accuracy figure quite concrete.

  2. The paper is also careful about which users influenced development, what information calibration could access, and what remains untested. Table 3 gives a practical account of what would be needed to advance each claim. The individual statistical tools are established, but the documented audit and its implications for this system are worth reporting.

Critical weaknesses
  1. The matched label-workflow comparison does not by itself isolate the causal effect of seeing the model prediction.** Holding videos and model outputs fixed is valuable, but the blind condition also introduces two raters and arbitration, and the manuscript does not establish that rater identity, scoring instructions, and adjudication were otherwise matched to the historical workflow. Thus, the 0.913-to-0.391 difference supports sensitivity to the reference-production process; attributing its magnitude specifically to anchoring requires additional controls. Appendix A.6's statement that model-visible agreement is an “upper bound on agreement with any independent reference” is also stronger than the evidence supports: exposure can inflate agreement, but does not mathematically impose such a bound. This matters because reference independence is a central empirical pillar of the contract.

  2. The refusal audit does not fully specify the reference and eligible populations needed to assess its own evidence contract.** Appendix A.7 reports 311 repetitions with 218 judgeable cases for the frozen audit, but 281 judgeable repetitions for the threshold sweep. It does not clearly map the repetition-level error labels used for these results to the independently blinded H1 labels versus the historical reflow annotations. Consequently, it is unclear how to interpret the reported 0.137 risk and whether Figure 5's sweep uses a comparable reference and denominator. The paper also states that the production guard script and per-repetition landmarks are unavailable in the supplementary bundle (p. 10). The concern is not that historical audits require new clinical data, but that this particular headline audit needs an explicit sample/reference ledger to be independently interpretable.

Questions for authors
  1. Were the historical and blinded judgments made by the same raters with the same rubric? How were the historical video-level labels formed? If rater or adjudication procedures changed as well, please distinguish the workflow comparison from a causal estimate of anchoring.

  2. For Appendix A.7, please explain the 218 versus 281 judgeable counts and specify the reference-label source, exclusions and denominators for both the frozen audit and the sweep. In particular, were these repetition-level labels obtained blind to the model, and how do they relate to H1?

1 comment

Dear Reviewer 9Bp9,

Thank you for your detailed feedback. We appreciate your recognition of the value of reassessing agreement with predictions held fixed and spelling out the evidence needed for each claim. We have made targeted changes in response to your comments on the labeling workflows and refusal evaluation and plan to include them in the revised manuscript.


  1. W1 & Q1: Model-visible review vs. blind labeling. We held the predictions fixed for the same 23 videos. The historical video label was the majority class across repetition-level judgments made with the model output visible; it agreed with the prediction on 21/23 videos. For H1, two raters independently judged each video as a whole without seeing the model output, and a third adjudicated disagreements. The resulting consensus agreed with the prediction on 9/23 videos. Section 4.3 and Appendix A.6 will clarify that the workflows differed in the unit of judgment and adjudication, and that we did not verify whether the raters or scoring rubrics matched. The comparison therefore measures sensitivity to the labeling workflow, not the causal effect of seeing the model output. We have also removed the claim that model-visible agreement is an upper bound.
  2. W2 & Q2: Samples and references for refusal. The historical audit computes coverage over all 311 logged repetitions. Of 218 judgeable repetitions, 215 were accepted, and the system verdict disagreed with the review label on 16 of those. The reported risk of 0.137 is the average of users’ disagreement rates among accepted, judgeable repetitions, with each user weighted equally. In the threshold replay, 281 repetitions had saved pose data; of these, the 189 judgeable repetitions form the coverage denominator (127 bird-dog, 62 kneeling leg raise). At each threshold, disagreement is computed separately for each exercise over the judgeable repetitions accepted at that threshold. The kneeling leg raise decision threshold was −10° in the historical system and −3° in the replay. Both analyses use historical model-visible repetition reviews, not H1. Appendix A.7 will report the denominators and inclusion criteria side by side.
  3. Q3: Reproducibility materials. The current supplement does not include complete repetition-level refusal records. We will list the missing fields in the reproducibility statement. Subject to data permissions and anonymity requirements, we will prepare the accept/refuse decisions, scalar scores, reference labels, and pseudonymized user and video IDs alongside the sweep code, without releasing the videos or the app.

Thank you again for your careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.