acceptodds
Under review as a conference paper at ICLR 2027

When Is Reuse Superiority Licensed? Evidence Separation for Heterogeneous Offline Experience Reuse

Abstract

Comparisons of heterogeneous offline experience reuse often collapse distinct questions into a single statement that one method is “better”: average outcome, target-wise benefit, source-selection quality, and deployed behavior are treated as interchangeable evidence. We study when such comparative claims are warranted. We introduce an executable, claim-specific evidence-separation protocol that first specifies what “better” means—including the estimand, population, reference, measurement contract, comparison budget, uncertainty unit, deployment contract, and scope—and then invokes the evidence required by that claim. In controlled finite-MDP problems with known answers, stronger audit configurations reduce wrong directional claims at matched claimed-decision coverage, while a critic-local forced-reuse score shows weak fine-grained agreement with exact downstream consequences but retains coarse consequence information. The result is a claim- licensing methodology for heterogeneous reuse, not a universal controller or a validated prospective harm predictor.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, heterogeneous experience reuse, evaluation methodology, evidence separation, claim licensing, negative transfer, uncertainty-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_neyHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about when comparative claims in heterogeneous offline experience reuse are actually licensed by the evidence. It splits claims into five types—mean outcome, target-wise benefit, selector quality, mechanism attribution, deployment performance—and ties each to an evidence contract. An audit then returns SUPPORTED, CONDITIONAL, UNRESOLVED, or NOT SUPPORTED. There's a finite-MDP benchmark with exact answers where the implemented audit configurations are compared at matched claimed-decision coverage, and a separate experiment on whether a critic-local degradation score tracks exact downstream policy consequences. Meta-World, ManiSkill3, and D4RL serve as case studies. The contribution the authors claim is methodological discipline, not a better reuse controller or a validated harm predictor, and I think that framing is accurate.

Critical strengths
  1. The evidence separation is operationally useful, not merely terminological. The paper connects different claims to concrete controls: paired target-level comparisons, matched reuse budgets, information-matched architectural controls, and direct deployment evaluation. These distinctions actually change which conclusions survive. The Exact-Simple-A control is a good example: it challenges the necessity of the complex architecture under the matched-information contract. The deployment studies similarly prevent a process-metric advantage from being promoted automatically to policy superiority. This makes the methodology useful for evaluating reuse methods, even if the broader protocol still needs sharper demonstration.
  2. The controlled validation directly measures an important failure mode and handles dependence carefully. At 20% claimed-decision coverage, P3 makes 8 opposite-direction errors among 1,440 issued claims, compared with 45 for P0/P1 and 31 for P2. Resampling keeps the six regimes of each problem together. The paper also reports the coverage ceiling and an unfavorable same-information control rather than claiming universal optimality. These choices give credible evidence of bounded utility and make the limits visible.
Critical weaknesses
  1. The additional benefit of P3 is not sufficiently separated from changes in information and decision rules. P1→P2 does isolate adding Q1 under the same r1 ranking. But P2→P3 simultaneously adds a second measurement, requires Q2 and sign agreement, and changes the ranking to min(r1, r2). Its gain therefore cannot be attributed to a single additional evidence check. The same-information control makes the issue clearer: at 2,593 claims it makes 5 directional errors versus P3’s 103, but it admits 181 illegal-unit and 515 support-shift items, so the admissibility contracts differ. I accept the reported configuration-level gain and the authors’ acknowledgment of this limitation. Still, clearer component comparisons would establish which design choices account for the additional benefit and make the audit design easier to reuse.
  2. The practical gain from unifying the evaluation principles is less clearly demonstrated than their explanatory value. Section 3 and Table 13 provide a procedure and concrete claim mappings, and the case studies convincingly show that different evidence types are not interchangeable. However, the current validation says less about which additional consequential judgments the unified workflow changes, or how it improves their consistency and reliability, beyond specifying estimands, using paired inference, and applying matched controls. The controlled local-severity comparison only partially addresses this question. My concern is the incremental value of the workflow as a reusable methodological contribution, not the absence of a procedure or a requirement to calibrate all four statuses universally.
Questions for authors
  1. P1 vs P2 cleanly isolates the addition of Q1, but the step from P2 to P3 simultaneously changes the measurements used, the qualification requirements, the sign-agreement condition, and the ranking rule. Among the 14 rules retained in Appendix B.5, which ones actually identify the contribution of each added component? I would like the gains obtained under the same information and the same admissible pool separated from gains that arise merely by changing which items are eligible. That distinction is central to understanding what the proposed audit design really adds.
  2. How were the narrower scopes that qualify for a CONDITIONAL judgment selected, and were they fixed before the results were examined? Even when restricted to evaluated scopes, there may still be a choice among several contracts or populations. Given that the underlying intervals are pointwise, what statistical interpretation should the final CONDITIONAL status carry, and how is that selection accounted for? (Section 3; Table 13.)
  3. Were the A0/A′ ordering correlations in Appendix E.2 based on directly evaluated intervention losses or on learned auditor predictions? If predictions were used, what evidence separates sensitivity to the measurement definition from error in estimating the ON/OFF differences? As E.3 itself notes, a high prediction-target correlation does not by itself establish an accurate ranking of those differences. This distinction matters for how the measurement-sensitivity result should be interpreted.
  4. Appendix G.1 says that the per-event accepted-mask lineage is incomplete. What exactly is missing, and which reported ManiSkill3 results can still be reconstructed from the available records? In particular, do the missing records affect the process contrasts or confidence intervals in Equations (31)–(32)? Please identify which conclusions remain independently checkable and which depend on the missing information.
Reviewer_VrCJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the question of "what kind of evidence supports the superiority of a method" in heterogeneous offline experience reuse. It proposes an evidence separation framework for different claims and analyzes this framework through controlled experiments and multiple reinforcement learning scenarios. The research emphasizes that evidence at different levels cannot be substituted for one another, providing a reference for evaluating experience reuse methods. However, current experiments primarily support the local effectiveness of some audit configurations. The novelty of this work is questionable.

Critical strengths
  1. The distinction between the evidence required for different claims is relatively clear.
  2. Negative results and experimental limitations are adequately disclosed.
Critical weaknesses
  1. The core methodology is not very innovative.
  2. The effectiveness of the complete audit framework has not yet been adequately validated. If the vast majority of comparative studies fail to reach definitive conclusions, how usable is such an auditing system in real-world research workflows?
Questions for authors
  1. What is fundamentally new beyond established statistical evaluation principles?
  2. Can the authors demonstrate prospective utility by applying a frozen audit protocol to new heterogeneous-reuse comparisons and evaluating its ability to identify scientifically reliable conclusions?
Reviewer_ub5aOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper proposes a claim-specific evidence-separation protocol for comparative claims about heterogeneous offline experience reuse. Each claim is specified and classified into one of five claim types. The protocol then checks it against the corresponding evidence contract and returns 'supported', 'conditional', 'unresolved', or 'not supported'.
In known-answer finite MDP experiments, stricter audit configurations reduce opposite-direction claims at matched claimed-decision coverage. Case studies on Meta-World, ManiSkill3, and D4RL, mainly auditing the authors’ RCER methods, show that different types of evidence can license different conclusions.

Critical strengths
  • The paper is explicit about what each piece of evidence does and does not license. It separates reuse coverage from claimed-decision coverage, avoids cross-panel comparisons, does inference at the appropriate independence unit, and does not treat CIs crossing zero as equivalence. It also reports findings that limit its own claims: a simpler same-information control outperforms P3, and the process-to-deployment bridge remains unresolved. This transparency is commendable.
  • Comparing directional error at matched claimed-decision coverage controls for the trivial error reduction that comes from abstaining more. The principle is sound and applicable beyond this setting.
Critical weaknesses
  • The controlled validation does not establish the reliability of the proposed protocol. Section 4 evaluates P0–P3, which are ranking and qualification configurations, not the full protocol. Status assignment across claim types and the narrowing procedure are never tested. The gains are also hard to attribute to the qualification design. The thresholds are tuned on the same generator family and target failure modes already present in the stress regimes, so robustness to unseen failure modes is uncertain. A stronger validation would build known-answer claims of different types, including overly broad claims with valid narrower alternatives, and evaluate the full procedure.

  • The statistical validity of CONDITIONAL narrowing is underspecified. If narrower claims are selected after inspecting results, pointwise intervals do not give valid inference for the selected claim, and restricting narrowing to evaluated claims does not remove this concern. The paper does not specify how the admissible narrower claims are determined, whether they are fixed in advance, or how selection and multiplicity are handled. This matters because scope control is the protocol’s central objective.

Questions for authors

see above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.