acceptodds
Under review as a conference paper at ICLR 2027

Historical Evaluation Without a Unique Past: Identification, Complexity, and Evidence from a Real-Time Archive

Abstract

Historical evaluation often uses the archive available today to approximate the data available at a past decision. Even when the decision protocol is known, incomplete provenance can leave several historical states consistent with it. We represent this ambiguity by a set of admissible worlds. A loss, loss contrast, or winner is identified when it is invariant across that set, even if the historical state itself remains unknown. We prove that strict winner reversal is NP-complete even for one cutoff, binary availability, fixed targets, frozen linear predictors, and squared loss. Ordered revisions admit an exact cutoff quotient, while fixed-dimensional affine timing permits exact evaluation over hyperplane-arrangement faces. An audit of 38 quarterly decisions using 1,180 official ALFRED snapshots shows why the admissible set matters. When one feed age is shared across feeds and fixed across decisions, same-world comparisons certify all ten pairwise rankings on a declared four-age support; separate marginal ranges leave four unresolved. Adding intermediate daily ages exposes one reversal, and allowing feed-specific fixed ages exposes two. If ages may instead vary by decision, only four rankings remain certified. Reusing identical resolved states reduces 256 worlds with fixed feed-specific ages to 32 states and cuts model calls eightfold. These results connect historical claim identification to explicit temporal assumptions, exact computational structure, and verifiable archival evidence.

Primary area:
learning theory
Keywords:
historical evaluation, partial identification, temporal data provenance, computational complexity, real-time data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jr1EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies retrospective evaluation when surviving evidence does not uniquely determine the historical state available at a past decision. It formalizes the ambiguity through an admissible-world set W(C,E)W(C,E), induced by a decision contract and surviving evidence, and defines a historical evaluation claim as identified when it is invariant across all admissible worlds.

A central methodological point is to evaluate competing models within the same historical world rather than constructing separate marginal performance ranges. The resulting paired contrast interval is always contained within the marginal interval and can therefore certify model rankings that would otherwise remain unresolved.

The theoretical analysis proves that strict historical winner reversal is NP-complete even under a restricted setup with binary availability and frozen linear predictors. The authors then identify exact tractable structure for ordered revision chains and fixed-dimensional affine timing.

Empirically, the paper audits 38 quarterly forecasting decisions using official ALFRED vintages. It progressively relaxes assumptions about shared feed age, available ages, cross-feed coupling, and cross-decision coupling, showing that model-ranking certification depends substantially on these temporal assumptions.

Critical strengths
  • The admissible-world formulation provides a clean way to distinguish uncertainty about historical state from uncertainty about the downstream evaluation claim that actually matters.

  • The hierarchy from state identification through loss, contrast, and winner identification is conceptually useful because a ranking can remain identifiable even when the exact historical state is not.

  • Same-world comparisons preserve dependence between models and yield strictly tighter contrast bounds than separate marginal ranges in relevant cases without requiring additional model evaluations.

  • The theory goes beyond worst-case hardness by identifying structured regimes where exact evaluation is possible, making the contribution more useful than an impossibility result alone.

  • The ALFRED audit concretely demonstrates sensitivity to temporal assumptions: certification falls from ten rankings to four as the admissible timing model becomes progressively less restrictive.

  • Exact state reuse is practically relevant: 256 feed-specific worlds collapse to 32 distinct resolved states, reducing model calls eightfold in the corresponding audit.

Critical weaknesses
  • The empirical study is narrow, covering four macroeconomic feeds, 38 pre-pandemic decisions, five small frozen predictors, and one target vintage.

  • The admissible-world set is itself assumption-dependent. Certification can therefore appear strong or weak depending on how aggressively the analyst constrains plausible ingestion timing.

  • The audit uses controlled schedule ambiguity rather than reconstructing genuinely undocumented historical production behavior, so it primarily demonstrates the framework rather than full forensic recovery.

  • Several tractability results depend on restrictive structure, including fixed timing dimension, ordered revisions, path-independent resolution, or casewise additive losses.

  • The empirical experiments use small predictors. The framework is architecture-independent in principle, but its computational practicality for expensive foundation-model evaluations is demonstrated only indirectly through state reuse.

Questions for authors

My main question concerns construction of the admissible set in practice. When exact ingestion logs are missing, what evidence or principles should determine whether assumptions such as shared feed age or fixed age across decisions are defensible? Since certification changes substantially when these assumptions are relaxed, practical guidance here seems central.

Second, can the authors say more about how they would handle uncertainty over the decision contract itself rather than only uncertainty within a fixed contract? For example, if transformation lineage or resolver behavior is only partially documented, would the natural extension be a union over contracts, and how would that affect certification?

Third, have the authors considered evaluating the framework with a larger or more nonlinear frozen model? I understand that the theoretical interface is model-agnostic, but an empirical example involving a more expensive evaluator would help establish how important the proposed state-reuse reductions become in modern ML settings.

Finally, how should practitioners choose between reporting the full claim image and its interval hull? The paper notes that interior interval values need not be attainable, so I would appreciate guidance on when the interval abstraction becomes materially misleading.

3 comments

We thank the reviewer for recognizing claim-specific identification and same-world comparisons. The revision adds evidence-based construction, uncertain-contract semantics, larger frozen-evaluator validation, and attainable-value reporting guidance.

Q1. What evidence supports the admissible worlds and their timing assumptions?

We distinguish archived vintages from operational events. Under a declared polling, completion, initialization, and source-ordering contract, we construct exactly the resolved inputs of all compatible complete histories. Missing records remain unknown; every returned input has a compatible-history witness. We prove both inclusions and verify them through independent enumeration and hidden-truth isolation (§4.6; Appendix D.4).

Shared ages across feeds or decisions require supporting operational evidence. With real ALFRED payloads and 200 semi-synthetic histories, 20% log retention still yields 7.15/10 certified comparisons on average under the correct contract.

With an incorrect two-day cap, retained evidence detects inconsistency in 111/200 histories, generating empty supports. The remaining 89 supports yield 8 false assertions among 811. The construction and its failure boundary are verified under the simulated contract; real production coverage needs independent operational evidence.

Q2. How can the framework represent uncertainty in the contract itself?

We retain the candidate contract as part of the uncertain object:

Ω = {(C,w): C∈𝒞, w∈W(C,E)}.

For a comparable claim G(C,w), its identified image is the union of contract-specific images. Its endpoints are the infimum of contract-specific lower endpoints and supremum of upper endpoints, reducing to minimum and maximum for finite attained families (Appendix D.3). This gives a direct rule across candidate contract interpretations.

Candidates must share the estimand, loss unit, and comparable case/evaluator semantics, with one candidate covering the realized contract and history. The formulation represents uncertain resolution and lineage without selecting one arbitrary contract. Different targets require distinct claims. This is a conditional theoretical extension.

Q3. Is the interface validated for nonlinear and expensive frozen evaluators?

Yes. We fit HGB and MLP once, freeze their parameters, and report all 84 seven-model pair–support intervals across the four supports (§4.8; Appendix F.2). Their measured RMSEs and the MLP convergence warning are retained, keeping applicability distinct from predictive superiority.

We also validate exact effective-input reuse with a frozen Qwen2.5-7B Transformer. Over the full feed-specific fixed-age support, 9,728 case-level requests correspond to 324 distinct effective inputs. On the prespecified measured subset, 608 requests reduce to 187 generation calls. Six paired naive/cached repetitions verify token-level agreement for all restored outputs (§4.8; Table 4; Appendix D.1).

The original 256→32 (8×) grouping concerns complete 38-case matrices, while Qwen uses case-level effective requests. These units cannot be combined. The experiment verifies exact reuse for a larger evaluator; timing remains diagnostic, and potential pretraining exposure limits forecasting claims.

Q4. When is the interval hull sufficient, and when should the attainable set be reported?

For shared ages {0,7,14,30}, the four attainable AR(1)–ridge MSE contrasts are approximately {−0.006305, −0.172430, −0.216736, −0.215396}. Their interval hull is [−0.216736, −0.006305]. The endpoints are attained, but most values between them are not (Appendix D.5; Table 12).

For a uniform strict-winner claim on finite nonempty support, exact extrema suffice. A hull may include values never produced by any historical world. Claims about a tie, an exact contrast value, or gaps require the attainable set or compatible-history witnesses; a hull spanning zero alone does not establish an attainable tie.

Thanks for the clarification. I have updated my rating accordingly.

Thank you very much for carefully considering our clarification and updating your assessment. We sincerely appreciate your time.

Reviewer_WixiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats historical (backtest) evaluation under incomplete provenance as a partial identification problem. A decision contract plus surviving evidence defines a set of admissible worlds, and a claim (a loss, a loss contrast, a winner) counts as certified if it’s the same in every admissible world. Theory: paired same-world intervals are no wider than marginal ones (Prop. 1), strict winner reversal is NP-complete (Thm. 1), and ordered revisions and affine timing admit exact reductions (Thms. 2, 3). The experiment uses ALFRED vintages for four US macro series over 38 quarterly decisions and five small frozen forecasters. Certified pairwise rankings go from 10 to 9, 8 or 4 depending on how the schedule set is declared, and caching identical states cuts model calls 8x.

Critical strengths

Strength 1. Good sounds for the framing. Separating “is the protocol valid” from “is the state identified” from “is the winner identified” is a nice way to think about backtests on revised data, and the takeaway (report the claim image, not just a point replay) is practical. The ALFRED setup is easy to reproduce.

Critical weaknesses

Weakness 1. The “certified” rankings don’t mean much in this audit. The admissible set is invented by the authors (Sec. 4.1 says the schedule restriction “is not inferred from ALFRED”), and the headline result of 10/10 certified only holds on the sparse {0,7,14,30} grid. Add days 1 to 6 and one ranking flips, let ages vary by decision and only 4 survive. So the method is very sensitive to a choice the paper gives no guidance on (to be fair, they don’t hide it). On top of that, the certified pairs are often statistically meaningless. AR(1) vs ridge is certified with a point contrast of 0.006 MSE (RMSE 1.756 vs 1.758) and keeps its sign in only about 51% of bootstrap draws (Table 7). And the historical mean beats every model, so I’m not convinced certifying rankings among the losers on 38 cases helps anyone.

Weakness 2. The technical contributions are fairly light. Prop. 1 is basically the observation that inf of a difference is at least the difference of inf and sup. The X3C reduction is the obvious one (shared binary choices over cases encode exact cover), Thm. 3 is standard arrangement enumeration, and the decision-varying case is just casewise separability of an additive loss. The 8x caching gain mostly comes from CPI never changing and payrolls changing only at 30 days (Table 5), which is a property of the data, not of the method. Nothing wrong, just thin for ICLR, and the transformer link in Sec. 5 is untested.

Questions for authors

Q1. How should a practitioner pick the admissible support in a real setting where nobody imposes it by design?

Q2. Would you consider only reporting a ranking as meaningful when it’s both historically certified and stable under resampling? How many of the 10 survive that?

2 comments

We thank the reviewer for focusing on support justification, statistical stability, and computational novelty. The revision adds an evidence-based construction, complete stability counts, and an explicit account of the theoretical and implemented contributions.

  1. How should practitioners choose admissible supports?

The original {0,7,14,30} grid is a declared contract, not an ALFRED-inferred schedule. We locate its exact breakdown: adding a one-day snapshot age makes AR(1)–ridge reversible, with paired MSE bounds [−0.006305, 0.003653]. The original 10/10 certifications apply to that sparse support; the revision reports breakdown thresholds under broader timing assumptions (§4.5; Appendix F.3).

We also construct supports from retained operational evidence. Under explicit polling, completion, initialization, and ordering rules, we compute the resolved-input image of all compatible histories, prove both set inclusions, and validate them by independent enumeration (§4.6; Appendix D.4). Missing logs remain unknown; timing coupling requires operational justification.

With real ALFRED payloads and semi-synthetic logs for 200 histories, 20% retention still certifies 7.15/10 comparisons on average under the correct contract. Fresh-point replay reverses 52/2,000 pair–history signs (2.6%), all AR(1)–ridge, across 52 histories. Evidence rejects an incorrect two-day age cap in 111/200 histories, leaving empty supports. Of 811 assertions from the other 89 nonempty supports, eight are false. This tests evidence-driven certification and contract failure, conditional on the simulated operating rules.

  1. How many certified rankings remain stable under resampling?

Under S4 with four-quarter moving-block resampling, 4/10 certified rankings retain their strict sign in at least 80% of draws, and 1/10 in at least 90%. Historical Mean–ridge reaches 95.50%; AR(1)–ridge reaches 50.85%. Under decision-varying timing (V4), 1/4 reaches 80% (Historical Mean–ridge: 92.85%; §4.7; Tables 27 and 30). These are descriptive resampling frequencies, not confidence levels.

Complete-model analysis adds a decision consequence. Historical Mean is uniformly best on S4, 31 fixed shared daily ages (S31), and feed-specific fixed ages (F4). Under V4, no model is uniformly best, and KNN becomes possibly optimal. Historical Mean remains the minimax-regret choice (worst-case regret 0.153008 MSE; Table 3). The optimality guarantee changes, while the minimax recommendation remains unchanged.

  1. What is new beyond standard mathematical tools and data-dependent caching?

Our claim-specific identification-and-computation framework shows why a protocol can leave historical states unresolved while identifying their model comparison. It formalizes the hierarchy of states, joint losses, contrasts, and winners, specifying the information each evaluation claim requires (§§1–2).

Theorem 1 proves NP-completeness of strict winner reversal from historical ambiguity alone, even with one cutoff, binary availability, fixed targets, frozen linear predictors, and no training. X3C is the reduction tool; the new result isolates hardness before learning. Complementarily, ordered revisions admit cutoff classes and fixed-dimensional affine timing admits evaluation-constant faces under explicit resolver assumptions (§3). These establish the problem’s computational boundary.

Proposition 1 establishes the benefit of preserving dependence: on S4, same-world evaluation certifies four additional rankings using the same model calls as marginal evaluation (§4.3).

The 256→32 (8×) complete-matrix grouping is data-dependent: CPI is unchanged and payroll features change only at 30 days on this grid. We also implement and independently validate a restricted p=1 temporal partition. Including discovery, it is slower here than ordinary input caching (0.698 vs 0.229 s; §4.8; Appendix E.2).

The frozen-Transformer interface in original §5 is now tested with a 7B evaluator: exact case-level reuse reduces 608 requests to 187 generation calls, with token-level equality across six paired runs (§4.8; Table 4). This case-level quotient differs from complete-matrix reuse. The experiment establishes exact output reuse and fewer generation calls, while the retained timings have a defined diagnostic scope.

We thank the reviewer again for the constructive comments. We have provided additional evidence on support justification and statistical stability, and clarified the paper’s theoretical and computational contributions in the revision. We would appreciate any further feedback on whether these changes address your main concerns.

Reviewer_aFYuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies historical model evaluation when retained evidence does not determine a unique past input state. A decision contract and retained evidence define admissible historical worlds, and the paper asks which losses, pairwise contrasts, and rankings remain identifiable across those worlds. It shows why comparing models within the same world can yield tighter bounds than combining separate loss ranges, establishes hardness for general timing uncertainty, and develops exact reductions under structured assumptions. An ALFRED audit uses 1,180 snapshots, four macroeconomic feeds, and 38 quarterly decisions. It demonstrates sensitivity to admissible timing assumptions, distinguishes historical identification from sampling uncertainty, and measures savings from reusing identical resolved states.

Critical strengths
  1. The paper gives a coherent identification framework with meaningful computational boundaries. It distinguishes the information needed to recover states from that needed to certify a comparison, and connects this distinction to hardness and exact structural reductions. This is a substantive contribution beyond a general warning about temporal leakage.
  2. The empirical audit tests the assumptions that determine the certificates. Expanding the age support, decoupling feeds, and allowing decision-specific timing change the number of certified rankings from ten to nine, eight, and four. Together with the sampling analysis, these results show precisely what a certificate does and does not establish.
Critical weaknesses
  1. The most consequential practical step is supplied by the audit design. Section 4.1 imposes the allowed feed ages and their coupling rather than deriving them from incomplete operational evidence. Yet a certificate depends on these choices covering the realized history. The paper explains how to evaluate a given uncertainty set, but does not demonstrate how an auditor can construct and justify that set from incomplete records. This leaves a substantial gap between the formal framework and its intended use.
  2. The empirical evidence establishes an illustrative case rather than a convincing level of practical impact. Four macroeconomic feeds, 38 decisions, and five small frozen predictors show that schedule assumptions can change rankings, but do not establish how often this ambiguity affects real model-selection decisions. The measured acceleration also comes from finite-state reuse, not the more general affine-timing algorithm. The broader operational value and computational scalability remain insufficiently demonstrated.
Questions for authors
  1. Section 4.1 states that the schedule restrictions are imposed by the audit design. Is there an audit case in which the admissible set is derived from retained operational evidence instead? How is coverage of the realized history justified in that case?
  2. What evidence supports the practical prevalence or decision impact of this ambiguity beyond the controlled ALFRED construction? In particular, is there an actual model-evaluation decision whose conclusion changes or becomes certifiable using the framework?
  3. Have the ordered-revision or affine-timing reductions been implemented beyond the finite-state reuse experiment? If so, what are their candidate counts and total runtimes, including class or arrangement construction?
2 comments

We thank the reviewer for recognizing the distinction between reconstructing a historical state and identifying an evaluation claim. The revision now provides a constructive evidence-to-support procedure, concrete evaluation consequences, and an implemented temporal reduction.

Q1. How can an auditor construct admissible supports and justify coverage?

We construct the support directly from retained evidence under an explicit event contract specifying polling, completion, initialization, and source-ordering rules. Missing log entries remain unknown events. The resulting support is exactly the set of resolved inputs generated by complete histories compatible with the retained records. We establish both inclusions, construct compatible-history witnesses, and verify the procedure through independent enumeration and hidden-truth isolation (§4.6; Appendix D.4).

Using real ALFRED payloads and semi-synthetic ingestion logs, we evaluate 200 histories under five levels of evidence retention. With only 20% of log records retained, the correct-contract procedure still certifies 7.15 of ten comparisons on average. A fresh-point reconstruction instead reverses the true sign in 52 of 2,000 pair–history comparisons (2.6%), all involving AR(1)–ridge, across 52 of the 200 histories.

The audit also detects incompatible operating assumptions. At 20% retention, an intentionally incorrect two-day age cap conflicts with retained evidence in 111/200 histories, producing empty supports and no certificates. The other 89 histories yield nonempty supports, with 8 false assertions among 811 strict assertions. This separates detected inconsistencies from residual failures under a misspecified contract. The coverage result is conditional on the declared operating contract; ALFRED records historical vintages, not an actual system’s ingestion logs.

Q2. Does the framework change evaluation conclusions or decisions?

Yes. A concrete later-vintage diagnostic reverses the AR(1)–AR(4) ordering: the historically admissible evaluation favors AR(4), whereas replaying the fixed 2026-08-21 vintage favors AR(1) (§4.2). The latter is a separate post-hoc comparison, outside the declared historical support.

We now identify the precise support assumptions behind such conclusions. The original four-age support certifies all ten pairwise rankings, yet admitting a one-day snapshot age makes AR(1)–ridge reversible, with a paired MSE contrast of approximately [−0.006305, 0.003653] (§4.5; Appendix F.3). The shared four-age (S4), shared daily-age (S31), feed-specific fixed-age (F4), and decision-varying shared-age (V4) supports certify 10/9/8/4 comparisons. The latter three are different relaxations, not a nested sequence.

The complete five-model analysis reveals a further consequence. Historical Mean is uniformly best on the four-age shared, daily shared, and feed-specific fixed-age supports. With decision-varying shared ages, no model remains uniformly best, and KNN joins Historical Mean in the possible-best set. Historical Mean remains the minimax-regret choice, with worst-case regret 0.153008 MSE. Thus the optimality guarantee changes even when the minimax recommendation does not (§4.7; Table 3; Appendix F.4). These are concrete evaluation consequences within the studied macroeconomic task.

Q3. Have the temporal structural reductions been implemented, and at what cost?

We implement a restricted fixed-p=1 shared-age partition, constructed from time predicates before model prediction. On the declared mathematical domain d∈[0,30], it yields 29 breakpoints, 57 singleton/open faces, and 29 active-version signatures, followed by a separate grouping into 26 numerical input matrices. Independent raw-timeline checks and endpoint witnesses verify the exact representation (Appendix E.2).

The six-repeat, discovery-inclusive median runtimes on the same 31-point grid are 0.237 s for direct enumeration, 0.229 s for input caching, and 0.698 s for temporal partitioning plus caching. The implementation establishes exact structural coverage, while its setup overhead makes it slower on this small instance. The continuous age variable is a mathematical extension of daily archive evidence, not a claim of observed intraday availability. General p>1 and ordered-revision implementations remain beyond this experiment.

We also validate reuse for a frozen 7B Transformer evaluator: on a fixed measured subset, exact effective-input grouping reduces 608 requests to 187 generation calls, with identical output tokens across six paired runs (§4.8; Table 4). This case-level reduction is distinct from the original 256→32 complete-matrix grouping. We report verified call savings without extrapolating a general wall-clock speedup.

We thank the reviewer again for the thoughtful feedback. We have addressed the main concerns with an evidence-to-support construction, concrete model-selection consequences, and an independently validated temporal implementation. We would greatly appreciate any further thoughts on whether these additions address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.