- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies retrospective evaluation when surviving evidence does not uniquely determine the historical state available at a past decision. It formalizes the ambiguity through an admissible-world set , induced by a decision contract and surviving evidence, and defines a historical evaluation claim as identified when it is invariant across all admissible worlds.
A central methodological point is to evaluate competing models within the same historical world rather than constructing separate marginal performance ranges. The resulting paired contrast interval is always contained within the marginal interval and can therefore certify model rankings that would otherwise remain unresolved.
The theoretical analysis proves that strict historical winner reversal is NP-complete even under a restricted setup with binary availability and frozen linear predictors. The authors then identify exact tractable structure for ordered revision chains and fixed-dimensional affine timing.
Empirically, the paper audits 38 quarterly forecasting decisions using official ALFRED vintages. It progressively relaxes assumptions about shared feed age, available ages, cross-feed coupling, and cross-decision coupling, showing that model-ranking certification depends substantially on these temporal assumptions.
Critical strengths
-
The admissible-world formulation provides a clean way to distinguish uncertainty about historical state from uncertainty about the downstream evaluation claim that actually matters.
-
The hierarchy from state identification through loss, contrast, and winner identification is conceptually useful because a ranking can remain identifiable even when the exact historical state is not.
-
Same-world comparisons preserve dependence between models and yield strictly tighter contrast bounds than separate marginal ranges in relevant cases without requiring additional model evaluations.
-
The theory goes beyond worst-case hardness by identifying structured regimes where exact evaluation is possible, making the contribution more useful than an impossibility result alone.
-
The ALFRED audit concretely demonstrates sensitivity to temporal assumptions: certification falls from ten rankings to four as the admissible timing model becomes progressively less restrictive.
-
Exact state reuse is practically relevant: 256 feed-specific worlds collapse to 32 distinct resolved states, reducing model calls eightfold in the corresponding audit.
Critical weaknesses
-
The empirical study is narrow, covering four macroeconomic feeds, 38 pre-pandemic decisions, five small frozen predictors, and one target vintage.
-
The admissible-world set is itself assumption-dependent. Certification can therefore appear strong or weak depending on how aggressively the analyst constrains plausible ingestion timing.
-
The audit uses controlled schedule ambiguity rather than reconstructing genuinely undocumented historical production behavior, so it primarily demonstrates the framework rather than full forensic recovery.
-
Several tractability results depend on restrictive structure, including fixed timing dimension, ordered revisions, path-independent resolution, or casewise additive losses.
-
The empirical experiments use small predictors. The framework is architecture-independent in principle, but its computational practicality for expensive foundation-model evaluations is demonstrated only indirectly through state reuse.
Questions for authors
My main question concerns construction of the admissible set in practice. When exact ingestion logs are missing, what evidence or principles should determine whether assumptions such as shared feed age or fixed age across decisions are defensible? Since certification changes substantially when these assumptions are relaxed, practical guidance here seems central.
Second, can the authors say more about how they would handle uncertainty over the decision contract itself rather than only uncertainty within a fixed contract? For example, if transformation lineage or resolver behavior is only partially documented, would the natural extension be a union over contracts, and how would that affect certification?
Third, have the authors considered evaluating the framework with a larger or more nonlinear frozen model? I understand that the theoretical interface is model-agnostic, but an empirical example involving a more expensive evaluator would help establish how important the proposed state-reuse reductions become in modern ML settings.
Finally, how should practitioners choose between reporting the full claim image and its interval hull? The paper notes that interior interval values need not be attainable, so I would appreciate guidance on when the interval abstraction becomes materially misleading.
3 comments
Thanks for the clarification. I have updated my rating accordingly.
Thank you very much for carefully considering our clarification and updating your assessment. We sincerely appreciate your time.
We thank the reviewer for recognizing claim-specific identification and same-world comparisons. The revision adds evidence-based construction, uncertain-contract semantics, larger frozen-evaluator validation, and attainable-value reporting guidance.
Q1. What evidence supports the admissible worlds and their timing assumptions?
We distinguish archived vintages from operational events. Under a declared polling, completion, initialization, and source-ordering contract, we construct exactly the resolved inputs of all compatible complete histories. Missing records remain unknown; every returned input has a compatible-history witness. We prove both inclusions and verify them through independent enumeration and hidden-truth isolation (§4.6; Appendix D.4).
Shared ages across feeds or decisions require supporting operational evidence. With real ALFRED payloads and 200 semi-synthetic histories, 20% log retention still yields 7.15/10 certified comparisons on average under the correct contract.
With an incorrect two-day cap, retained evidence detects inconsistency in 111/200 histories, generating empty supports. The remaining 89 supports yield 8 false assertions among 811. The construction and its failure boundary are verified under the simulated contract; real production coverage needs independent operational evidence.
Q2. How can the framework represent uncertainty in the contract itself?
We retain the candidate contract as part of the uncertain object:
Ω = {(C,w): C∈𝒞, w∈W(C,E)}.
For a comparable claim G(C,w), its identified image is the union of contract-specific images. Its endpoints are the infimum of contract-specific lower endpoints and supremum of upper endpoints, reducing to minimum and maximum for finite attained families (Appendix D.3). This gives a direct rule across candidate contract interpretations.
Candidates must share the estimand, loss unit, and comparable case/evaluator semantics, with one candidate covering the realized contract and history. The formulation represents uncertain resolution and lineage without selecting one arbitrary contract. Different targets require distinct claims. This is a conditional theoretical extension.
Q3. Is the interface validated for nonlinear and expensive frozen evaluators?
Yes. We fit HGB and MLP once, freeze their parameters, and report all 84 seven-model pair–support intervals across the four supports (§4.8; Appendix F.2). Their measured RMSEs and the MLP convergence warning are retained, keeping applicability distinct from predictive superiority.
We also validate exact effective-input reuse with a frozen Qwen2.5-7B Transformer. Over the full feed-specific fixed-age support, 9,728 case-level requests correspond to 324 distinct effective inputs. On the prespecified measured subset, 608 requests reduce to 187 generation calls. Six paired naive/cached repetitions verify token-level agreement for all restored outputs (§4.8; Table 4; Appendix D.1).
The original 256→32 (8×) grouping concerns complete 38-case matrices, while Qwen uses case-level effective requests. These units cannot be combined. The experiment verifies exact reuse for a larger evaluator; timing remains diagnostic, and potential pretraining exposure limits forecasting claims.
Q4. When is the interval hull sufficient, and when should the attainable set be reported?
For shared ages {0,7,14,30}, the four attainable AR(1)–ridge MSE contrasts are approximately {−0.006305, −0.172430, −0.216736, −0.215396}. Their interval hull is [−0.216736, −0.006305]. The endpoints are attained, but most values between them are not (Appendix D.5; Table 12).
For a uniform strict-winner claim on finite nonempty support, exact extrema suffice. A hull may include values never produced by any historical world. Claims about a tie, an exact contrast value, or gaps require the attainable set or compatible-history witnesses; a hull spanning zero alone does not establish an attainable tie.