acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Evidence Under Change: A Categorical Semantics for Decision-Relative Reuse

Abstract

Evaluation results often outlive the procedures that produced them, and when tasks, verifiers, environments or reporting rules change, what an old archive still warrants depends on the question asked of it. We give a finite categorical semantics in which evaluation settings and changes form a category, typed compatibility relations carry retained evidence to possible target records and each downstream use observes them through a decision profile. Tracking record multiplicity and use dependence separates exact reuse from decision-sufficient multiplicity, unresolved disagreement, empty compatibility and invalid comparison. Since changes com- pose, alternative routes can be compared through the profile sets they induce, and any discrepancy yields a finite witness of decision-relevant non-transport. For a fixed family of future observations we characterize the coarsest quotient preserving them and when compressed interfaces preserve propagation exactly. In exploratory case studies, a coarse CI archive still settles release but loses triage detail, eICU report contracts agree at current thresholds yet separate at every prespecified split- decile, and a corrected DeepSWE v1-to-v1.1 re-evaluation finds no qualifying same-state use-specific sufficiency witness. The semantics shows what historical evidence still supports for a declared use, where it falls short and which distinctions an archive must keep.

Primary area:
foundation or frontier models, including LLMs
Keywords:
category theory, evaluation reuse, transportability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xXBDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when an archived evaluation can support a declared decision after the evaluation setting changes. It formalizes settings and their modifications using a finite set of categories, propagates potential records through compatibility relations, and interprets the resulting records via application-specific decision rules. This framework yields five reuse grades, a test for consistency across alternative change paths, and criteria for preserving information required for future use.

Critical strengths
  1. The paper connects sequential propagation, alternative change paths, and future retention in one formal account (section 3.5–5). Its examples make clear why an intermediate grade cannot replace the underlying possible-record set, and why matching grade labels do not establish path agreement.

  2. The paper clearly separates what an old report can and cannot tell us. In the CI study, the same report settles the release decision but cannot identify which test suites need attention (Table 2; Appendix B.2).

Critical weaknesses

W1. Section 2 acknowledges that relational propagation, observational quotients, and view preservation have precedents. The main results in section 4 and section 5 are largely direct consequences of set images, an observation-signature quotient, and constancy on interface fibres. The paper should compare its theoretic results with relational semantics and observational completeness, explaining which result or evaluation question these prior approaches do not already capture.

W2. The conclusions are contingent on application-supplied relations (RR), as acknowledged in section 3.4 and 7. In the eICU route study, the observed agreement at decile boundaries and disagreement at split-deciles follow directly from the selected fine and coarse report contracts (Appendix B.3). To clarify what the case study establishes beyond the formal construction, the paper should demonstrate how the grades vary under alternative defensible contracts and explain which contracts align with actual reporting practices.

Questions for authors
  1. Section 2 cites prior results on observational completeness and database view preservation. What do Theorems 5.1 and 5.2 add to these results in their assumptions or conclusions? A direct comparison would clarify the theoretical contribution.

  2. After the evaluation setting changes, do any of your case studies show that the same old report can answer one question but not another? If not, please clarify what each case study shows.

  3. Work on incomplete databases (Certain Answers as Objects and Knowledge) seems to be a particularly relevant comparison that is absent from the bibliography.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xXBD (1/2): relation to prior results (W1, Q1, Q3)

Thank you for a precise review. Your Sections 4, 5 and 7 are now Sections 5, 6 and 8, and Appendix B is now Appendix C.

W1 and Q1. What do Theorems 6.1 and 6.2 add?

Section 2 acknowledges that relational propagation, observational quotients, and view preservation have precedents. The main results in section 4 and section 5 are largely direct consequences of set images, an observation-signature quotient, and constancy on interface fibres. The paper should compare its theoretic results with relational semantics and observational completeness, explaining which result or evaluation question these prior approaches do not already capture.

Section 2 cites prior results on observational completeness and database view preservation. What do Theorems 5.1 and 5.2 add to these results in their assumptions or conclusions? A direct comparison would clarify the theoretical contribution.

What they add is in the assumptions and the conclusions.

  • Assumptions: observations are indexed by use, and each is the set of decision profiles a record reaches through a declared relation, with the empty set as a value, so the quotient separates "no compatible continuation" from "answers disagree".
  • Conclusions: "coarsest" is a universal property relative to a finite declared family of continuation-and-use pairs, and when descent fails Theorem 6.2 returns two source records the interface merges and a target interface state reachable from only one, naming what an archive must keep. Descent checked on each declared change then holds along every path (Corollary 6.3, added in this revision and stated in Section 6.2).

The constructions themselves are known, as the paper states (Section 2; Section 6.2, which now says what our statements change; Table 5, Appendix I): Theorem 6.1 is the kernel of the joint observation map, as in coalgebraic behavioural equivalence and automata minimization (Silva et al., Colcombet and Petrişan) and observational completeness of abstract domains (Amato and Scozzari), and Theorem 6.2 is constancy on fibres, as in view determinacy (Fan et al., Benedikt et al.). Against observational completeness, our observations are indexed by use and taken once after propagation, not along an iterated computation (Table 5).

Relational semantics gives the propagation as the standard direct-image functor (Section 3.3). What the paper adds on top of it are three readings evaluation needs: a separate grade of one possibility set per use, empty and invalid comparisons kept as grades (where Dempster's rule would normalize conflict away), and two declared routes compared after observation, with a finite witness of disagreement (Proposition 5.3). Each occurs in the studies (Tables 1 and 2): on SWE-ABS one archive settles 64 of 273 bar decisions and none of 741 pair orders, five of 109 software source records have an empty image and both setup stops are invalid, and the eICU routes separate at all ten split deciles. The main formal result new in the revision is the sampled grade (Proposition 4.3), which goes beyond the worst-case grade: it certifies decisions jointly at a stated level from a random sample of the open items, provided the declared relation holds.

Q3. Incomplete databases.

Work on incomplete databases (Certain Answers as Objects and Knowledge) seems to be a particularly relevant comparison that is absent from the bibliography.

Section 2 and Table 5 (Appendix I) now draw this closest comparison. A possibility set plays the role of the possible worlds of an incomplete database (Imieliński and Lipski, 1984). Libkin (2016) defines a certain answer as the greatest lower bound of a query's answers over the possible worlds, in an information ordering. For a flat order on a use's answers, with a bottom for disagreement and an adjoined top for an empty set of worlds, this greatest lower bound over the possibility set is our transport value, and its bottom and top are reported as the grades "diagnostic" and "empty". The "exact" grade counts worlds and has no certain-answer counterpart.

AuthorsOpenReview ↗

Response to Reviewer xXBD (2/2): differentiated use (Q2) and dependence on the supplied relations (W2)

Q2. Does an old report answer one question but not another after a change?

After the evaluation setting changes, do any of your case studies show that the same old report can answer one question but not another? If not, please clarify what each case study shows.

Yes, in Table 1 under real changes, as Section 4 states. One per-instance archive settles 64 of 273 bar decisions and 0 of 741 pair orders on SWE-ABS, 2 of 175 and 0 of 300 on SWE-bench Verified to Lite, and 263 of 459 against 504 of 11,628 on MMBench. In Table 2, the CI and eICU marginal studies show the split under the identity change only, the eICU route study shows two routes leaving different answers possible, and the software study cannot, as each task has one fully determined record and every use is exact.

W2. The eICU results follow from the chosen contracts.

The conclusions are contingent on application-supplied relations (R), as acknowledged in section 3.4 and 7. In the eICU route study, the observed agreement at decile boundaries and disagreement at split-deciles follow directly from the selected fine and coarse report contracts (Appendix B.3). To clarify what the case study establishes beyond the formal construction, the paper should demonstrate how the grades vary under alternative defensible contracts and explain which contracts align with actual reporting practices.

The study shows the audit at work on the public eICU Demo cohort: from two declared contracts it computes agreement at the current deciles and separation at every split decile, with a finite witness at each (Proposition 5.3). These are consequences of the contracts, as Section 7 now states, and Table 3 (Appendix C.3) shows how they vary with the contract. It derives the grades analytically under five contracts on the same 1,683 stays (801 under 65, 882 aged 65 or more): a one-point risk histogram, 5-point blocks, deciles, quintiles and age-stratified deciles. The flagged count C at a threshold is settled exactly at a block boundary and inside a block of D stays takes D + 1 values. The count aged 65 or more is settled at a boundary only under the age-stratified contract, and under the other four ranges over [max(0, C − 801), min(882, C)], open whenever 0 < C < 1,683. All 9 current-decile thresholds are block boundaries except under quintiles (4 of 9), and all 10 split-decile thresholds only under the one-point and 5-point contracts. Calibration reports for ICU severity scores commonly group stays by predicted risk. The decile contract uses the fixed cut points of the Hosmer-Lemeshow Ĥ statistic (risk bands of ten points), while the more widely used Ĉ statistic groups stays by deciles of predicted risk (Lemeshow and Hosmer, 1982; Appendix C). A calibration table does not by itself fix which contract a report follows, and we have no evidence of which contract any hospital uses, so Table 3 gives the grades under each of the five (Appendix C.3).

In Table 1 each relation is read off the published change documentation, or reconstructed from the records for MMBench, and the new per-item records test it where they can contradict it. The relations of ImageNet ReaL and RePOPE are the published relabeling lists. Those of HAL and SWE-ABS cannot be contradicted by the new records used (HAL's regrade replays only ungraded episodes, and SWE-ABS reran only old passes), so their zero wrong answers do not test the relation. If HAL's graded episodes are also replayed, its relation breaks and the worst-case grade is wrong on 10 pair orders and 3 bar decisions (Appendix G). Identity on kept items fails on SWE-bench Verified to Lite and MMBench, where the level does not apply (Table 1, Appendix E), and on the former, in a separate 400-draw check (Appendix E.3), the observed new score falls outside some system's joint interval in 0.26 (25%) and 0.80 (50%) of draws. MMBench also depends on the systems included (Appendix G, Section 8). On the 153 whose new records pass a reliability check, the per-item record settles 504 of 11,628 pair orders with 0 wrong. With all 220, 257 pair orders are wrong, because some excluded systems' new records are mostly unparseable and so are not what the relation assumes. The same check on old records only, which a user has before the change, keeps 159 systems and settles 525 with 2 wrong.

Based on my understanding, this paper appears to apply relational algebra from database theory to the field of AI, which I find somewhat puzzling given my limited familiarity with this specific intersection. It seems the authors are using abstract algebra to describe data flows within an AI system. However, it is unclear what specific "AI system" refers to in this context. Does this framework offer practical guidance for actual neural network training, considering the authors focus on frontier models? It would be greatly appreciated if the authors could provide concrete examples or a simplified toy example to translate the complex algebraic formulations into practical applications.

Reviewer_7H6MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper is concerned with the problem of the evaluation of evidence over time, after some changes have taken place. The question under consideration is to what extent the original information can still be used, what it can be used for, under what circumstances, and under what declared changes. The author(s) propose to make sense of this through a categorical approach. The papers contributions are:

i) A categorical semantics for the reuse of evidence. ii) A setting in which possibility sets compose. iii) The statement of two retention results. iv) The statement of a probabilistic outer bound.

Critical strengths

The paper deals with an interesting and important question, What is a proper framework for evaluating evidence under change?, and provides a formal structured response (or rather, sets of responses to the relevant sub-questions) through the provision of a categorical semantics.

Critical weaknesses

I can appreciate the importance of the question posed in the title of the paper. Also, I'm able to follow the paper in broad strokes, and my assessment is that there is a tangible, non-trivial contribution here. However, the paper is highly technical and there is, in my view, not sufficient detail to place this in context for interested and semi-informed outsiders, such as myself. For a conference such as ICLR, I think it would have been a good idea to attempt to make the paper a bit more accessible for the non-specialist.

Questions for authors

I don't have specific questions for the author(s).

3 comments

Hi everyone,

I don't see any author responses to the questions and comments of any of the reviewers. The other reviews also don't change my view of the paper substantially. I am therefore keeping my scores unchanged.

Thank you for your attention and patience. We will post responses and an updated manuscript very soon.

AuthorsOpenReview ↗

Response to Reviewer 7H6M

[...] However, the paper is highly technical and there is, in my view, not sufficient detail to place this in context for interested and semi-informed outsiders, such as myself. For a conference such as ICLR, I think it would have been a good idea to attempt to make the paper a bit more accessible for the non-specialist.

Thank you for judging that there is "a tangible, non-trivial contribution". The revision adds a way in for readers outside the area while keeping every result of the submission and its formal structure.

Before any notation, Section 1 now names four results reused after a change, from a leaderboard whose automated judge is replaced to a clinical risk score reported at new thresholds, and works a CI example: a kept failing-test count (10 of 16) leaves 8,008 possible records, all saying "hold the release" but giving six answers on which suites are clean, and one more kept field leaves one record. Section 2 places the work against evaluation methodology, evidence theory and incomplete databases, and quotient and minimization constructions. Section 3.1 says what each part of the categorical semantics does and that each law is a finite check on sets and relations. Terms are glossed at first use and long sentences are split.

Appendix A, now first among the appendices and pointed to from Section 1, gives a plain-language glossary of 26 terms and a primer that defines each categorical notion the paper uses with its reading here. Each entry explains the term in plain words, gives a CI or SWE-bench instance where one helps and names the section that defines it.

Section 4 now reads one row of Table 1 in plain steps. SWE-ABS regrades 39 agents on SWE-bench Verified under strengthened tests, where a pass may become a fail and never the reverse, so each agent's new score lies between zero and its old one. Their kept per-task record therefore settles 64 of 273 bar questions (an agent below a bar stays below it) and none of 741 pair questions. Re-running a random quarter of the open tasks, at 18.5% of a full re-run's cost, settles on average 148.9 bar and 4.9 pair questions, all correct together with probability at least 0.95, provided the declared relation holds.

On soundness, every proof is given in full (Section 5, Appendices D and E), and Appendix B lists the finite check each categorical statement unfolds to.

Reviewer_p9FkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of determining when old evaluation results can still be trusted after the evaluation setup changes, for example when a new grader or a new benchmark version becomes available. The user must provide the old results, rules for how old results map to possible new results, and the decisions they want to make. The proposed framework lists all possible new results the old data could correspond to and checks whether they all lead to the same final decision. If so, the old results can be reused; otherwise, a rerun is needed. The framework also flags when two different ways of applying a change give different answers. The paper illustrates the framework with a set of small exploratory case studies.

The case studies are quite small and exploratory, the computational tractability is unaddressed, and the case for why this belongs at ICLR rather than a formal methods or philosophy-of-science venue is not made. The paper would be considerably stronger with one or two realistic-scale case studies where the framework provides real value by showing that historical evaluations can (or cannot) be reused, and with a direct comparison to what existing evaluation methodology recommendations already provide.

Critical strengths

S1. I find the framing of reuse as decision-specific, where old data is assessed with respect to a specific downstream decision, a useful approach compared to an all-or-nothing view. It allows the user to reuse old data whenever it is sufficient for a downstream task, without rerunning the evaluation.

S2. The authors are transparent about the limitations of the paper.

Critical weaknesses

W1. The paper opens with category-theoretic definitions before any motivating example, which makes it difficult to understand. I'm saying this as a reviewer with knowledge of category theory. Most readers will not have taken a graduate course category theory. Furthermore, the paper uses long and complex sentences, which adds to the difficulty. Leading with a concrete and substantive example would greatly improve the readability of the paper and the understandability of the proposed framework.

W2. It is not clear to me what category theory contributes beyond notation. The use of the categorical machinery is not well motivated in the paper.

W3. The method checks reuse by listing every possible result and testing the decision on each. A single 16-test report already gives |U| = 8,008 possibilities, as reported in Section 6. The paper offers no efficient algorithm or complexity analysis, so it is unclear whether the method can be applied beyond small cases.

W4. The paper does not show that the framework helps in practice. It lacks a comparison with how results are currently reused, and it reports no measured benefit, such as avoided wrong decisions or saved rerun costs.

W5. The framework expects a lot from the user. The user must specify the settings and changes, a compatibility rule for every change, complete record definitions, decision rules for every use, and the list of future questions in advance. There is not a guidance on how to build or justify these inputs.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer p9Fk (1/3): readability and category theory

Thank you for a close reading. Your summary asks for realistic-scale case studies showing that old evaluations can or cannot be reused, and for a direct comparison with existing evaluation-methodology recommendations (our comment 3/3). Section 4 and Table 1 now grade six public benchmark changes (up to 169 systems and 14,196 pair decisions per change) beside a fixed 5-point margin, a rule of thumb without guarantee, shown for comparison. On the five Table 1 rows where the stated level applies (three assume count-only archives), the margin carries forward 811 of 1,683 bar decisions, 107 of them wrong. The worst-case grade settles 337, none wrong. Re-running a random 25% of the open items, at 3.6% to 18.5% of a full re-run, settles 1,196 on average, none wrong in 200 draws, provided the declared relation holds. For pair orders the margin settles more than the sample (5,687 against 4,292), with 131 wrong. This comment answers W1 and W2, the next W3 and W4, and the third W5, scale, venue and prior recommendations.

W1. Example first, shorter sentences.

The paper opens with category-theoretic definitions before any motivating example, [...]

The second paragraph of Section 1 now works this example, before any notation:

For a small example, take sixteen tests in four suites of 3, 3, 4 and 6 tests, and a CI archive that kept only the total number of failing tests, 10. The complete records still possible are the 8,008 placements of 10 failures among 16 tests [...]. A release gate that holds if any test failed answers "hold" on all of them [...], whereas suite triage (which suites are clean) takes six different answers [...].

The same split appears under a real change in Section 4: under the stricter tests of SWE-ABS, one per-instance archive settles 64 of 273 bar decisions and none of 741 pair orders. In the default case the category is the free category on a finite acyclic graph of settings, with one relation on each edge, and a route is a path (Section 3.1). Terms are glossed at first use, Appendix A opens the appendices with a plain-language glossary of 26 terms and a primer that defines each categorical notion the paper uses (category, relations, functor, natural transformation, quotient, support of a kernel) with its reading here, and long sentences are split.

W2. What category theory contributes.

It is not clear to me what category theory contributes beyond notation.

Every categorical statement in the paper unfolds to a finite check on sets and relations, and Appendix B lists that reading line by line. What the categorical form contributes is the specification: declared changes are the syntax, the functor R into relations is their interpretation, and its laws say which checks a user owes (Section 3.1). It fixes four things.

  1. What composes. Possibility sets propagate step by step without loss, and the result does not depend on how a path is written (the functor law), while grades have no composition law (Proposition 3.1), so an archive must carry the set and not its grade.
  2. What may be identified. Two routes between the same settings stay distinct arrows unless a path equation is declared. A declared equation that the relations violate must be repaired or dropped, and routes without one are audited for each use (Section 5).
  3. What a local check buys. Descent (Theorem 6.2) is a commuting square of relations, and squares paste: once it holds for each generating change it holds along every path, so a whole network of changes is propagated on the retained interfaces, which form a natural transformation (Corollary 6.3, added in this revision and stated in Section 6.2).
  4. What transfers. The support functor from kernels to relations carries the grade, witness, quotient and descent statements to every probabilistic refinement (Appendix F).

"Coarsest" in Theorem 6.1 is a universal property. The proofs use these definitions and need nothing beyond them: Corollary 6.3 pastes squares in Rel and uses that R is a functor, Proposition F.2 uses that the support map and every refinement preserve composition and that kernels on the generators of a free category extend uniquely, and Lemma 5.2, on which the sampled grade's nesting rests, is the order on relations. The limits are as precise: on a free category, our default, the functor laws hold by construction and bind only where a path equation is declared, and each law is a finite check on sets and relations, plus finite probabilities for the sampled grade and Appendix F (Appendix B).

For evaluation reuse the semantics delivers three things: empty and invalid comparisons are reported outcomes, the same archive is graded separately for each use, and parallel routes are audited with a witness. The quotient and descent constructions it uses are known (Sections 2 and 6.2, Appendix I). The revision's other additions, the sampled grade (Proposition 4.3) and the measurements, are in our next two comments.

AuthorsOpenReview ↗

Response to Reviewer p9Fk (2/3): tractability and measured benefit

W3. Tractability.

[...] The paper offers no efficient algorithm or complexity analysis, so it is unclear whether the method can be applied beyond small cases.

Structured inputs are graded without listing records (Section 3.4, Appendix H). For threshold or order decisions on sums the possible new scores form a closed-form interval, which grades Table 1 (up to 14,196 pair decisions per change), and the CI example is graded on 60 count vectors, not 8,008 records. These cases cover Table 1 and the CI studies. In general, for succinct records and rules with polynomial-time checks, deciding whether all compatible records give one profile is coNP-complete (Appendix H).

At scale, Section 3.4 grades 3,002 runs of a Kubernetes end-to-end job (90.7 days, about 1,280 tests in 12 suites per run) from the total failure count alone, under the identity change. Although a failing run leaves up to 10^101.3 possible records, every grade for every run took 14.6 s, including brute-force checks of the closed forms on small instances. The count settles the hold decision for all 3,002 runs, but of the 36,024 per-suite paging decisions it settles only the 33,660 trivial ones in passing runs and none of the 2,364 in the 197 failing runs, while per-suite counts settle all 36,024.

W4. Comparison with a margin rule, and measured benefit.

The paper does not show that the framework helps in practice. It lacks a comparison with how results are currently reused, [...]

Table 1 now sets the grades beside a fixed 5-point margin, a rule of thumb without guarantee, shown for comparison: it carries an old pair order or bar verdict forward when the old gap or distance to the bar exceeds 5 points, and needs no new data. On a count archive the worst-case grade is itself a rule of this form, with margins for each system that the declared relation justifies in place of a fixed number of points (Section 4). Wrong answers in every column are counted against the observed new scores. Sampled cells are means over 200 random draws at familywise level 0.05 (the level does not apply to MMBench or Verified→Lite, where identity on kept items fails), 25% is of the open part, and Cost is the sample's item runs as a share of a full re-run. Totals (pairs · bars) follow each name.

Settled/wrong, pairs · bars5-pt marginWorst caseSampled 25%Cost
MMBench v1.0→v1.1 (11,628 · 459)8,055/0 · 322/0504/0 · 263/06,275.2/0 · 367.9/06.7%
ImageNet ReaL, count-only (2,211 · 201)1,019/0 · 94/13172/0 · 14/01,164.4/0 · 188/07.6%
SWE-bench Verified→Lite (300 · 175)255/0 · 144/80/0 · 2/0130.7/0 · 109.1/0.0117.3%
RePOPE accuracy, count-only (14,196 · 507)1,606/14 · 183/4253/0 · 102/01,647.3/0 · 398.6/03.6%
RePOPE F1, count-only (14,196 · 507)2,354/30 · 182/52/0 · 8/01,104/0 · 305.1/03.6%
HAL tau-bench airline (741 · 195)597/87 · 158/17342/0 · 149/0371.7/0 · 155.5/04.1%
SWE-ABS (741 · 273)111/0 · 194/680/0 · 64/04.9/0 · 148.9/018.5%

Avoided wrong decisions: the margin is wrong somewhere in six of the seven rows, on 131 pair orders and 115 bar decisions in all, and settles more where a change is benign. These are 13 bars on ImageNet ReaL, 8 bars on Verified→Lite, 14 pairs and 4 bars on RePOPE accuracy, 30 and 5 on RePOPE F1, 87 and 17 on HAL (23 of the 87 are pairs that the new records tie) and 68 bars on SWE-ABS. The worst-case grade announces no wrong answer in any row. This is the guarantee of Proposition 4.3 where the declared relation holds, and Table 1 measures what the guarantee costs in settled decisions, beside the margin's errors. The relation is the published relabeling list on ImageNet ReaL and RePOPE, holds by construction of the regrade on HAL, and cannot be contradicted by the public records on SWE-ABS, where only old passes were re-run (Appendix G). On MMBench the zero is for the 153 systems that pass a reliability check needing the new records.

Section 4 now sweeps the margin from 0.5 to 5 points over all 14 change-and-metric cases (Table 4, Appendix E.3). Every margin in that range is wrong in at least 5 of them for pairs and in at least 10 for bars. At 0.5 points, close to carrying every old verdict forward, it is wrong in 10 of the 14 for pairs and in all 14 for bars. The smallest margin error-free on all 14, 26.0 points for pairs and 30.0 for bars, is a ceiling known only after the new scores exist.

Saved re-run cost: a sample of 25% of the open part costs 3.6% to 18.5% of a full re-run. On average it settles 188 of 201 bar decisions on ImageNet ReaL (worst case 14), 398.6 of 507 on RePOPE accuracy (102) and 148.9 of 273 on SWE-ABS (64), three rows where the level applies, with no wrong announcement in any of the 200 draws, provided the declared relation holds. Decisions the sample leaves open go to a re-run, and savings in a deployed workflow are not measured (Section 8).

AuthorsOpenReview ↗

Response to Reviewer p9Fk (3/3): inputs, scale, venue and prior recommendations

W5. The burden of inputs.

The framework expects a lot from the user. The user must specify the settings and changes, a compatibility rule for every change, complete record definitions, decision rules for every use, and the list of future questions in advance. There is not a guidance on how to build or justify these inputs.

In the Table 1 studies every input was read off public documentation or records: the settings are benchmark versions, the records are per-item results, the uses are pair orders and bars, and the relation is the documented change. The main text now covers three points.

  1. A safe default (Section 5.1, after Lemma 5.2). Widening a declared relation only adds possible answers, so a decision settled under it keeps its answer under any narrower relation that leaves some record possible. A user unsure of a relation can declare one that certainly contains the true one and lose settled decisions, never correctness.
  2. The Table 1 relations (Section 4), read off the change documentation (or the records, for MMBench), were of four kinds: identity on kept items with dropped items removed and added ones open (MMBench, SWE-bench Verified to Lite), stricter-only, where a pass may become a fail and never the reverse (SWE-ABS), its converse looser-only (HAL), and a published relabeling footprint (ImageNet ReaL, RePOPE).
  3. Future questions need declaring only to compress an archive (Theorem 6.1, Appendix H). With complete records kept, any later decision can be graded.

Where new per-item records can contradict the declared relation, on which the guarantee is conditional, they test it. Identity on kept items fails on SWE-bench Verified to Lite and MMBench, whose kept items change on re-execution. Among the Table 1 changes these are the two whose new records come from re-running the systems, and the others regrade stored outputs. By the safe default of Section 5.1, a user who doubts identity on kept items declares them open and loses settled decisions, never correctness, and on these two re-executed changes the records bear that doubt out.

Realistic-scale case studies. Table 1 is at realistic scale on public per-item records (Appendix G), with 153 systems and 11,628 pair decisions on MMBench, 169 systems and 14,196 pairs on RePOPE, 67 models on ImageNet ReaL and 39 runs or agents each on HAL and SWE-ABS. Both outcomes you name occur (table in our previous comment). Reusable without a re-run: under the ImageNet label-error correction, which touches 631 of the 50,000 images, a count-only archive settles 1,627 of 2,211 pair orders and 165 of 201 bars. Under ReaL, which touches 14,284 images, it settles 172 and 14 (Table 4, Appendix E.3; Appendix G). Neither has a wrong answer against the new labels. HAL's per-record archive settles 342 of 741 and 149 of 195, with zero wrong by construction of the regrade. SWE-ABS's per-instance archive settles none of 741 pair orders, since the new tests replace every instance's test patch (Section 4 reads this row step by step).

Why ICLR. The paper studies revisions of evaluations of AI systems (automated judges, benchmark versions, agent verifiers, label corrections), which are costly to re-run and so routinely reused, the setting of the submission's primary area (foundation or frontier models). Every Table 1 change is between public versions, splits or regrades of a benchmark in routine use (ImageNet, POPE, MMBench, SWE-bench, tau-bench). Appendices B.1 and J.2 instantiate the semantics for Bayesian evaluation with an automated judge, item response theory and agent benchmarks.

Existing evaluation-methodology recommendations. Section 2 now states the difference. Evaluation cards, measurement frameworks and item-level reporting say what to record (intended uses, provenance, per-item results) but not whether, for a given archive, change and decision, the old result still settles that decision. The grade decides this from those records and a declared change relation. Table 1 shows both sides. Where per-item outputs are stored, as item-level reporting recommends, a label correction is settled exactly with no re-run (ImageNet ReaL, RePOPE; Appendix G). Where a change adds items or strengthens tests, per-item records leave most decisions open (none of 741 pair orders settled on SWE-ABS), and the sampled grade says what a partial re-run settles at a stated familywise level, provided the declared relation holds (the level does not apply to MMBench or SWE-bench Verified to Lite). Section 6 says which records a declared decision needs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for careful reviews and for the strengths they identify. Reviewer 7H6M finds the question "interesting and important" and sees "a tangible, non-trivial contribution". Reviewer p9Fk finds decision-specific reuse "a useful approach compared to an all-or-nothing view", one that lets a user "reuse old data whenever it is sufficient for a downstream task, without rerunning the evaluation", and the authors "transparent about the limitations". Reviewer xXBD finds that the paper "connects sequential propagation, alternative change paths, and future retention in one formal account", shows "why an intermediate grade cannot replace the underlying possible-record set" and "clearly separates what an old report can and cannot tell us". The revision keeps every result behind these strengths and adds what the reviews ask for. The key changes, by concern (revisions in dark green):

  • Practical value at scale (p9Fk W3, W4; xXBD Q2). Section 4 adds a sampled grade: re-run a uniformly random sample of the items a change leaves open and announce a decision only when it holds at a stated familywise level, provided the declared relation holds (Proposition 4.3; proofs and all 14 change-and-metric cases in Appendix E). On the five Table 1 rows where the stated level applies (three assume count-only archives), a fixed 5-point margin, a rule of thumb without guarantee, shown for comparison, carries forward 811 of 1,683 bar decisions, 107 of them wrong. The worst-case grade settles 337, none wrong. Re-running a random 25% of the open items, at 3.6% to 18.5% of the cost of a full re-run, settles 1,196 on average, none wrong in 200 draws, provided the declared relation holds. For pair orders the margin settles more than the sample (5,687 against 4,292), with 131 wrong. On SWE-ABS one archive settles 64 of 273 bar decisions and none of 741 pair orders, and Section 4 reads that row step by step. Feasibility is separate: Section 3.4 grades 3,002 runs of a Kubernetes CI job in 14.6 seconds, and Appendix H gives closed forms and coNP-completeness.
  • Accessibility (p9Fk W1, 7H6M). Section 1 works a CI example before any notation. Appendix A opens the appendices with a plain-language glossary and a primer on the category theory the paper uses, and the abstract ends in plain words.
  • What the formal results add (p9Fk W2; xXBD W1, Q1, Q3). Section 3.1 states the categorical semantics as a specification and says what each part does: the functor law makes propagation independent of how a path is written, Theorem 6.1's quotient is a universal property, descent squares paste so that a check on each change certifies every path (Corollary 6.3, new in Section 6.2), and the support functor carries every grade, witness, quotient and descent statement to probabilistic refinements (Appendix F). Section 6.2 says what Theorems 6.1 and 6.2 change in known constructions, Table 5 (Appendix I) compares each with its precedent, certain answers included, and Appendix B lists what each proof uses.
  • Supplied relations and contracts (p9Fk W5, xXBD W2). Section 4 names the four kinds of declared relation in Table 1 and where the new records contradict one, and Section 5.1 gives a safe default: declare a wider relation and lose settled decisions, never correctness. Table 3 (Appendix C.3) grades the eICU uses under five report contracts, and Section 7 states that the results follow from the contracts.
  • Disclosure. Section 8 and Appendices E and G say where Table 1 results need the new data or hold by construction, and that the sampled-grade analysis (level reading, interval method, headline sample fraction) was revised after its first results were seen. The new records show the declared relation failing on two of the six changes, and the paper says where.

Numbering, submission → revision: Definition 4.1, Lemma 4.2, Proposition 4.3 → 5.1, 5.2, 5.3; Theorems 5.1, 5.2 → 6.1, 6.2; Section 3.6 → Appendix F (Definition 3.2, Proposition 3.3 → F.1, F.2); the Section 4.1 example and its Table 1 → Appendix J.1 and Table 6; Table 2 → Appendix C. Definitions 4.1, 4.2 and Proposition 4.3 now state the new sampled grade. Case studies and limitations are Sections 7 and 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.