- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when an archived evaluation can support a declared decision after the evaluation setting changes. It formalizes settings and their modifications using a finite set of categories, propagates potential records through compatibility relations, and interprets the resulting records via application-specific decision rules. This framework yields five reuse grades, a test for consistency across alternative change paths, and criteria for preserving information required for future use.
Critical strengths
-
The paper connects sequential propagation, alternative change paths, and future retention in one formal account (section 3.5–5). Its examples make clear why an intermediate grade cannot replace the underlying possible-record set, and why matching grade labels do not establish path agreement.
-
The paper clearly separates what an old report can and cannot tell us. In the CI study, the same report settles the release decision but cannot identify which test suites need attention (Table 2; Appendix B.2).
Critical weaknesses
W1. Section 2 acknowledges that relational propagation, observational quotients, and view preservation have precedents. The main results in section 4 and section 5 are largely direct consequences of set images, an observation-signature quotient, and constancy on interface fibres. The paper should compare its theoretic results with relational semantics and observational completeness, explaining which result or evaluation question these prior approaches do not already capture.
W2. The conclusions are contingent on application-supplied relations (), as acknowledged in section 3.4 and 7. In the eICU route study, the observed agreement at decile boundaries and disagreement at split-deciles follow directly from the selected fine and coarse report contracts (Appendix B.3). To clarify what the case study establishes beyond the formal construction, the paper should demonstrate how the grades vary under alternative defensible contracts and explain which contracts align with actual reporting practices.
Questions for authors
-
Section 2 cites prior results on observational completeness and database view preservation. What do Theorems 5.1 and 5.2 add to these results in their assumptions or conclusions? A direct comparison would clarify the theoretical contribution.
-
After the evaluation setting changes, do any of your case studies show that the same old report can answer one question but not another? If not, please clarify what each case study shows.
-
Work on incomplete databases (Certain Answers as Objects and Knowledge) seems to be a particularly relevant comparison that is absent from the bibliography.
3 comments
Response to Reviewer xXBD (2/2): differentiated use (Q2) and dependence on the supplied relations (W2)
Q2. Does an old report answer one question but not another after a change?
After the evaluation setting changes, do any of your case studies show that the same old report can answer one question but not another? If not, please clarify what each case study shows.
Yes, in Table 1 under real changes, as Section 4 states. One per-instance archive settles 64 of 273 bar decisions and 0 of 741 pair orders on SWE-ABS, 2 of 175 and 0 of 300 on SWE-bench Verified to Lite, and 263 of 459 against 504 of 11,628 on MMBench. In Table 2, the CI and eICU marginal studies show the split under the identity change only, the eICU route study shows two routes leaving different answers possible, and the software study cannot, as each task has one fully determined record and every use is exact.
W2. The eICU results follow from the chosen contracts.
The conclusions are contingent on application-supplied relations (R), as acknowledged in section 3.4 and 7. In the eICU route study, the observed agreement at decile boundaries and disagreement at split-deciles follow directly from the selected fine and coarse report contracts (Appendix B.3). To clarify what the case study establishes beyond the formal construction, the paper should demonstrate how the grades vary under alternative defensible contracts and explain which contracts align with actual reporting practices.
The study shows the audit at work on the public eICU Demo cohort: from two declared contracts it computes agreement at the current deciles and separation at every split decile, with a finite witness at each (Proposition 5.3). These are consequences of the contracts, as Section 7 now states, and Table 3 (Appendix C.3) shows how they vary with the contract. It derives the grades analytically under five contracts on the same 1,683 stays (801 under 65, 882 aged 65 or more): a one-point risk histogram, 5-point blocks, deciles, quintiles and age-stratified deciles. The flagged count C at a threshold is settled exactly at a block boundary and inside a block of D stays takes D + 1 values. The count aged 65 or more is settled at a boundary only under the age-stratified contract, and under the other four ranges over [max(0, C − 801), min(882, C)], open whenever 0 < C < 1,683. All 9 current-decile thresholds are block boundaries except under quintiles (4 of 9), and all 10 split-decile thresholds only under the one-point and 5-point contracts. Calibration reports for ICU severity scores commonly group stays by predicted risk. The decile contract uses the fixed cut points of the Hosmer-Lemeshow Ĥ statistic (risk bands of ten points), while the more widely used Ĉ statistic groups stays by deciles of predicted risk (Lemeshow and Hosmer, 1982; Appendix C). A calibration table does not by itself fix which contract a report follows, and we have no evidence of which contract any hospital uses, so Table 3 gives the grades under each of the five (Appendix C.3).
In Table 1 each relation is read off the published change documentation, or reconstructed from the records for MMBench, and the new per-item records test it where they can contradict it. The relations of ImageNet ReaL and RePOPE are the published relabeling lists. Those of HAL and SWE-ABS cannot be contradicted by the new records used (HAL's regrade replays only ungraded episodes, and SWE-ABS reran only old passes), so their zero wrong answers do not test the relation. If HAL's graded episodes are also replayed, its relation breaks and the worst-case grade is wrong on 10 pair orders and 3 bar decisions (Appendix G). Identity on kept items fails on SWE-bench Verified to Lite and MMBench, where the level does not apply (Table 1, Appendix E), and on the former, in a separate 400-draw check (Appendix E.3), the observed new score falls outside some system's joint interval in 0.26 (25%) and 0.80 (50%) of draws. MMBench also depends on the systems included (Appendix G, Section 8). On the 153 whose new records pass a reliability check, the per-item record settles 504 of 11,628 pair orders with 0 wrong. With all 220, 257 pair orders are wrong, because some excluded systems' new records are mostly unparseable and so are not what the relation assumes. The same check on old records only, which a user has before the change, keeps 159 systems and settles 525 with 2 wrong.
Based on my understanding, this paper appears to apply relational algebra from database theory to the field of AI, which I find somewhat puzzling given my limited familiarity with this specific intersection. It seems the authors are using abstract algebra to describe data flows within an AI system. However, it is unclear what specific "AI system" refers to in this context. Does this framework offer practical guidance for actual neural network training, considering the authors focus on frontier models? It would be greatly appreciated if the authors could provide concrete examples or a simplified toy example to translate the complex algebraic formulations into practical applications.
Response to Reviewer xXBD (1/2): relation to prior results (W1, Q1, Q3)
Thank you for a precise review. Your Sections 4, 5 and 7 are now Sections 5, 6 and 8, and Appendix B is now Appendix C.
W1 and Q1. What do Theorems 6.1 and 6.2 add?
What they add is in the assumptions and the conclusions.
The constructions themselves are known, as the paper states (Section 2; Section 6.2, which now says what our statements change; Table 5, Appendix I): Theorem 6.1 is the kernel of the joint observation map, as in coalgebraic behavioural equivalence and automata minimization (Silva et al., Colcombet and Petrişan) and observational completeness of abstract domains (Amato and Scozzari), and Theorem 6.2 is constancy on fibres, as in view determinacy (Fan et al., Benedikt et al.). Against observational completeness, our observations are indexed by use and taken once after propagation, not along an iterated computation (Table 5).
Relational semantics gives the propagation as the standard direct-image functor (Section 3.3). What the paper adds on top of it are three readings evaluation needs: a separate grade of one possibility set per use, empty and invalid comparisons kept as grades (where Dempster's rule would normalize conflict away), and two declared routes compared after observation, with a finite witness of disagreement (Proposition 5.3). Each occurs in the studies (Tables 1 and 2): on SWE-ABS one archive settles 64 of 273 bar decisions and none of 741 pair orders, five of 109 software source records have an empty image and both setup stops are invalid, and the eICU routes separate at all ten split deciles. The main formal result new in the revision is the sampled grade (Proposition 4.3), which goes beyond the worst-case grade: it certifies decisions jointly at a stated level from a random sample of the open items, provided the declared relation holds.
Q3. Incomplete databases.
Section 2 and Table 5 (Appendix I) now draw this closest comparison. A possibility set plays the role of the possible worlds of an incomplete database (Imieliński and Lipski, 1984). Libkin (2016) defines a certain answer as the greatest lower bound of a query's answers over the possible worlds, in an information ordering. For a flat order on a use's answers, with a bottom for disagreement and an adjoined top for an empty set of worlds, this greatest lower bound over the possibility set is our transport value, and its bottom and top are reported as the grades "diagnostic" and "empty". The "exact" grade counts worlds and has no certain-answer counterpart.