- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is about when comparative claims in heterogeneous offline experience reuse are actually licensed by the evidence. It splits claims into five types—mean outcome, target-wise benefit, selector quality, mechanism attribution, deployment performance—and ties each to an evidence contract. An audit then returns SUPPORTED, CONDITIONAL, UNRESOLVED, or NOT SUPPORTED. There's a finite-MDP benchmark with exact answers where the implemented audit configurations are compared at matched claimed-decision coverage, and a separate experiment on whether a critic-local degradation score tracks exact downstream policy consequences. Meta-World, ManiSkill3, and D4RL serve as case studies. The contribution the authors claim is methodological discipline, not a better reuse controller or a validated harm predictor, and I think that framing is accurate.
Critical strengths
- The evidence separation is operationally useful, not merely terminological. The paper connects different claims to concrete controls: paired target-level comparisons, matched reuse budgets, information-matched architectural controls, and direct deployment evaluation. These distinctions actually change which conclusions survive. The Exact-Simple-A control is a good example: it challenges the necessity of the complex architecture under the matched-information contract. The deployment studies similarly prevent a process-metric advantage from being promoted automatically to policy superiority. This makes the methodology useful for evaluating reuse methods, even if the broader protocol still needs sharper demonstration.
- The controlled validation directly measures an important failure mode and handles dependence carefully. At 20% claimed-decision coverage, P3 makes 8 opposite-direction errors among 1,440 issued claims, compared with 45 for P0/P1 and 31 for P2. Resampling keeps the six regimes of each problem together. The paper also reports the coverage ceiling and an unfavorable same-information control rather than claiming universal optimality. These choices give credible evidence of bounded utility and make the limits visible.
Critical weaknesses
- The additional benefit of P3 is not sufficiently separated from changes in information and decision rules. P1→P2 does isolate adding Q1 under the same r1 ranking. But P2→P3 simultaneously adds a second measurement, requires Q2 and sign agreement, and changes the ranking to min(r1, r2). Its gain therefore cannot be attributed to a single additional evidence check. The same-information control makes the issue clearer: at 2,593 claims it makes 5 directional errors versus P3’s 103, but it admits 181 illegal-unit and 515 support-shift items, so the admissibility contracts differ. I accept the reported configuration-level gain and the authors’ acknowledgment of this limitation. Still, clearer component comparisons would establish which design choices account for the additional benefit and make the audit design easier to reuse.
- The practical gain from unifying the evaluation principles is less clearly demonstrated than their explanatory value. Section 3 and Table 13 provide a procedure and concrete claim mappings, and the case studies convincingly show that different evidence types are not interchangeable. However, the current validation says less about which additional consequential judgments the unified workflow changes, or how it improves their consistency and reliability, beyond specifying estimands, using paired inference, and applying matched controls. The controlled local-severity comparison only partially addresses this question. My concern is the incremental value of the workflow as a reusable methodological contribution, not the absence of a procedure or a requirement to calibrate all four statuses universally.
Questions for authors
- P1 vs P2 cleanly isolates the addition of Q1, but the step from P2 to P3 simultaneously changes the measurements used, the qualification requirements, the sign-agreement condition, and the ranking rule. Among the 14 rules retained in Appendix B.5, which ones actually identify the contribution of each added component? I would like the gains obtained under the same information and the same admissible pool separated from gains that arise merely by changing which items are eligible. That distinction is central to understanding what the proposed audit design really adds.
- How were the narrower scopes that qualify for a CONDITIONAL judgment selected, and were they fixed before the results were examined? Even when restricted to evaluated scopes, there may still be a choice among several contracts or populations. Given that the underlying intervals are pointwise, what statistical interpretation should the final CONDITIONAL status carry, and how is that selection accounted for? (Section 3; Table 13.)
- Were the A0/A′ ordering correlations in Appendix E.2 based on directly evaluated intervention losses or on learned auditor predictions? If predictions were used, what evidence separates sensitivity to the measurement definition from error in estimating the ON/OFF differences? As E.3 itself notes, a high prediction-target correlation does not by itself establish an accurate ranking of those differences. This distinction matters for how the measurement-sensitivity result should be interpreted.
- Appendix G.1 says that the per-event accepted-mask lineage is incomplete. What exactly is missing, and which reported ManiSkill3 results can still be reconstructed from the available records? In particular, do the missing records affect the process contrasts or confidence intervals in Equations (31)–(32)? Please identify which conclusions remain independently checkable and which depend on the missing information.