acceptodds
Under review as a conference paper at ICLR 2027

Identifiable Evaluation of Learned Systems: From Behavioral Collisions to Measurement Design

Abstract

Evaluation scores can be precise even when the protocol does not identify the behavioral property they are used to measure. We formulate protocol identifiability relative to a declared finite behavior class, an estimand, and a candidate measurement family. A protocol fails when two behaviors with different estimand values induce the same observable signature; such cross-estimand collisions are executable certificates of under-identification, and the collision structure itself defines a measurement-design problem: synthesize a minimum support that separates every declared alternative, or repair a failed protocol by minimum augmentation. The result is a constructive audit-to-repair workflow—expose the witness, add the cheapest distinguishing measurements, re-audit—assembled from classical hitting-set and concept-identification machinery. In a controlled language-model testbed with solver-verified labels, exhaustive audits show that base-only measurements collapse seven frozen policies, whereas a two-cell synthesized support separates them; on a fifty-policy stress class, certified synthesis identifies where 50,000 same-budget random supports all fail. Applied to classes derived from the models’ own predictions, a frozen 15-cell synthesized support identifies all 86 derivation signatures, while held-out re-audit exposes one residual collision and its one-cell repair; same-budget entropy and random supports leave substantially more unresolved alternatives. A confirmatory rerun with surface-matched target and sham edits then measures a positive base-conditioned target–sham asymmetry across four model families after controlling the prespecified edit-surface features. The framework turns a validity concern into an auditable procedure for measurement design; all empirical results are controlled-testbed consequences, not external validation.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, evaluation methodology, identifiability, measurement design, behavioral evaluation, counterfactual evaluation, protocol design, hitting set

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tpAzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether an eval protocol can discriminate between behaviors corresponding to different values of a target estimand. It gives the definition of identification over some finite behavior class, and returns a collision witness whenever 2 alternatives have different estimands but an identical observed response.

Measurement selection and repair are, thus, reduced to hitting-set problems over the measurements that differentiate pairs. Solver-labeled reasoning problems are used to create experiments with base, target, and sham worlds, multiple possible label mappings, and flexible response readouts.

For a class of 86 model derived signatures, a frozen 15 cell support resolves all derivation collisions and leaves a single held out collision which is resolved by an additional measurement. A separate matched control analysis finds higher sham preservation than target fidelity among base-correct responses. Limitations explicitly bound conclusions to the controlled testbed.

Critical strengths
  1. It turns a vague concern into something checkable: The phrase “This eval might not measure what it claims” is hard to act on - but here, when a protocol fails, the audit names the 2 behaviors it confuses + the measurements that'd separate them. Appendix C shows this pretty well: remove any single control condition and the audit points to a specific collision. As a good example: without target edits, a model that never updates looks identical to one that reasons correctly - so that gives designers a concrete reason for each control.

  2. The reporting is careful. The paper uses exhaustive enumeration where feasible, reports full random-baseline distributions, checks all 26 minimum designs, admits tie-break sensitivity and validates oracle labels with blinded human reviewers (k around 0.94) - I trust the numbers as a description of the testbed

Critical weaknesses
  1. The method is closer to existing work than the paper suggests: picking the fewest attributes that still separate every pair of differently-labeled items is a known problem - in rough set theory these are decision-relative reducts, which are computed as hitting sets over a discernibility matrix whose entries match the paper’s D {h,h’} sets (Pawlak , 1982, Skowron & Rauszer, 1992). Min features selection (Almuallim & Dietterich 1991) and minimum test collection (Garey & Johnson 1979) are fairly close as well. The paper claims novelty because it separates behaviors relative to an estimand (Section 2.3, Table 4) - but these methods already separate relative to labels - so the contribution is mainly an application of known tools, which then puts more weight on showing practical value.

  2. The experiments mostly confirm what was built into them: the 7 behaviors were designed so that each control is needed - and the larger synthetic sets were generated so that specific measurements become necessary - so random selection failing is expected and it tells us little about real protocols

The model-derived design only covers patterns already seen. In Appendix H shows it breaks once unseen patterns are added, with ~6.6k failures after 5k additions - and unseen behaviors are exactly the most important ones in this case. Simple greedy selection also comes within 1 pair of the exact solution (Table 2) - so exact synthesis doesn't look necessary

The model finding has a simple alternative explanation: in a sham edit, the correct answer stays the same - so a model that tends to repeat its earlier answer will look good on shams, and bad on targets. The paper doesn't rule this out and this analysis doesn't use the audit framework in any case. The models are also small models from 2024, with base accuracy near 0.5

To be convincing I think the paper needs at least 1 case where the audit finds a significant collision in an existing benchmark and changes how a published result should be read

Questions for authors
  1. How does your audit & synthesis procedure differ from decision-relative reducts in rough set theory or from min-features selection? Because both already find the smallest attribute set that separates every pair of differently labeled items, so if there's a property your setting handles that these methods don't - please state it clearly.

  2. Can you show 1 case where the audit finds a significant collision in an existing benchmark or published eval, and (most importantly) that that collision changes how a reported result should be read? I think a single convincing example would do much more for the paper’s significance than additional synthetic classes.

  3. In a sham edit, the correct answer stays the same - so a model that tends to repeat its base answer will score well on shams and poorly on targets. Among base-correct cases where the target answer is wrong: how often does it simply match the base answer? Does the asymmetry hold up once you account for this?

2 comments
AuthorsOpenReview ↗

Response to tpAz: benchmark consequence, classical scope, and persistence

Thank you for the careful review. We have added a public-benchmark collision and a frozen new-theory repair validation, while clarifying the classical-method correspondence and the limits of the matched-response analysis.

Locations: ARC witness, Appendix O.3 (p. 27); frozen repair, Tables 19–20 (p. 25); classical methods, Section 2.3/Appendix B; persistence, Table 23 (p. 28).

  1. Existing-benchmark consequence (W2/Q2). In released ARC-Challenge data, Amber-7B ckpt_090 and OLMo1-7B step461000 produce identical binary correctness on the same 12 native Fluid-selected items and the same partial-panel ability estimate, despite full scores of 437/1,172 and 551/1,172 (37.29% vs. 47.01%; a 9.73-point gap). Thus identical sparse observations do not identify equality of full ARC accuracy on this declared checkpoint class. One additional distinguishing item separates the pair. This changes how equality of these sparse observations can be interpreted, not a published aggregate conclusion. Fluid's approximate-ability objective is evaluated separately. Table 22/Figure 10 report exact-count collision and ambiguity curves, with discovery/calibration differences disclosed.

Beyond hand-designed H7–H50 stress cases, Appendix N evaluates a repair frozen before 96 new original-text ProofWriter theories (192 units). Complete-target recovery improves from 45/192 (original 12-cell pipeline) to 72/192 (repaired 17-cell pipeline); covered errors fall from 37 to 7, with 113 abstentions (Table 19). This compares complete pipelines, not a pure selector effect. Holding the enriched class/lookup fixed, adding five measurements improves recovery from 34/192 to 72/192. At equal 17-cell count, exact, coverage-greedy, conditional entropy and target information gain all recover 72/192, compared with a 32-seed random mean of 50.09/192 (Table 20). This equal-budget replay is secondary and retrospective, and effective heuristics can match exact synthesis.

Class-relative limitations remain: Appendix H shows that unseen alternatives can break a frozen certificate; unknown projections may abstain, but unseen behaviors can also match known projections incorrectly. The added Qwen3-8B panel does not establish broad natural-task or frontier-model transfer.

  1. Decision-relative reducts (W1/Q1). We agree that finite deterministic support selection is mathematically the same separation problem as decision-relative reduct/minimum-feature selection. Section 2.3 and Appendix B now state this correspondence, distinguish minimum-cardinality from inclusion-minimal solutions, and attribute the underlying mathematics to prior work. We do not claim a new hitting-set theorem. Our evaluation contribution is an executable audit-to-repair procedure: specify the estimand and alternatives; expose a cross-estimand collision when measurements are insufficient; identify the distinguishing measurements; and augment existing support. The ARC witness and frozen validation examine the usefulness of this operational workflow.

  2. Answer persistence (W3/Q3). We have quantified repetition rather than attributing asymmetry to a hidden mechanism. Among base-correct target errors, repetitions are 232/266 (balanced Llama3.1) and 89/198 (balanced Qwen2.5), versus 104/120 and 62/102 on the second source. Appendix P decomposes the sham-minus-target contrast on the original base-correct denominator: nonrepeat contributions are -3.72, +5.12, +7.69, and +12.21 points across those four model/source combinations (Table 23); a positive nonrepeat contribution is not universal.

This decomposition is descriptive, not a persistence-adjusted causal effect. Repetition is defined using the target response, so removing repeated errors or conditioning on nonrepetition selects on outcomes. Separate world queries do not eliminate a stable persistence tendency. We therefore restrict the matched result to an observed conditional contrast, separate from the class-relative identification certificates, and make no claim about an identified internal reasoning mechanism.

We hope the concrete ARC witness and independently frozen repair validation address the practical-value concern.

Thanks for the thorough response - the added experiments and narrower claims improve the paper, but my concerns about novelty and practical impact remain, so I am keeping my score

Reviewer_Ws6cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that an evaluation protocol may fail to distinguish behaviors that have different values of the target estimand but produce identical observable response patterns. The authors develop an operational workflow for auditing and repairing such protocols. The workflow identifies collision instances that demonstrate failures of identifiability, introduces the lowest-cost distinguishing measurements, and then reruns the identifiability audit. By formulating support-set selection as a hitting-set problem, the approach integrates protocol auditing, minimum-cost support-set synthesis, and incremental repair.

Critical strengths
  1. Clear problem formulation: The paper explicitly distinguishes the statistical precision of evaluation scores from the structural identifiability of the protocol. It further converts observational collisions between behaviors with different estimand values into checkable certificates, providing a clear formulation of the problem.

  2. Relatively systematic experimental analysis: In addition to comparing random and heuristic measurement selection under matched budgets, the authors examine support-set sizes across H7–H50, component necessity, model-derived signatures, and held-out re-auditing. The matched sham design also helps reduce certain confounding factors associated with predefined editing surface features, making the experimental analysis relatively systematic.

Critical weaknesses
  1. Insufficient validation of key claims: The main baselines consist primarily of greedy, entropy-based, and random selection strategies, with insufficient comparisons against recent approaches to cost-efficient evaluation or optimal measurement selection under a unified objective and realistic budget constraints. Moreover, in Table 2, coverage-greedy produces only one additional ambiguous cell compared with exact synthesis on the held-out evaluation. Therefore, the evidence supporting the claimed advantages remains limited.

  2. Limited model coverage: All four evaluated models are from 2024, and the experiments are conducted primarily on controlled three-valued logic tasks. The lack of evaluations using more recent models and independent naturalistic tasks limits the generalizability of the findings.

  3. Concerns about the reporting of key empirical results: The paper states that per-model estimates, confidence intervals, and permutation analyses for all four models are provided in Appendix I. However, the supplied PDF does not include the corresponding complete results in Appendix I. Additionally, the Joint|Base value of 0.487 reported in Table 1 cannot be obtained by rounding an integer number of successes divided by 444 to three decimal places. These issues raise concerns about the completeness and consistency of the reported empirical results.

Questions for authors

Q1: Are the per-model estimates, confidence intervals, and permutation analyses for all four models provided in the paper? Could the authors provide these complete results and discuss their implications?

Q2: Could the authors add comparable experiments against measurement-selection methods from the past two years, particularly those related to adaptive evaluation, capability estimation, or optimal experimental design, and report budget–performance curves using consistent collision or ambiguity metrics?

1 comment
AuthorsOpenReview ↗

Response to Ws6c: equal-budget comparison, native Fluid, and full statistics

Thank you for the comparison and reporting questions. The revision now places the added experiments directly in the manuscript with consistent targets and explicit locations.

Result locations: equal-budget repair, Table 20 (p. 25); native ability, Table 21 (p. 26); collision audit and curves, Table 22 and Figure 10 (p. 27); model-wise statistics, Tables 13–16 (pp. 20, 22).

  1. Practical role of selection at a fixed budget (W1).The workflow can audit and repair supports produced by exact or heuristic selectors. On the original bridge, exact identifies at 15 cells versus greedy's 16; held-out re-audit separates 46/48 versus 45/48 (Section 7.1). We retain that modest instance benefit. In the new equal-count repair replay, every method preserves the same 12 cells and adds five, using the same earlier class and lookup. Exact, coverage-greedy, conditional entropy and target information gain all recover 72/192 complete targets; marginal entropy recovers 45/192; 32 random supports average 50.09/192, range 35–68 (Table 20). This supports the usefulness of measurement choice and effective heuristics within the workflow. The replay is explicitly secondary and retrospective; equal counts do not imply equal token or monetary cost.

  2. Native recent method and a common audit endpoint (Q2). Appendix O runs the unmodified released Fluid implementation on its concrete-item ARC-Challenge/GSM8K banks, 452 checkpoints per task from six trajectories. At 12 items, ability MAE against the full-panel MAP reference is 0.1998/0.3145; 32-seed Random-IRT means are 0.4588/0.9447 (Table 21). These evaluate Fluid's native ability objective; they are not compared numerically to exact-identification rates.

To address consistent collision metrics, we separately audit native adaptive transcripts and class-fitted static/greedy supports for the same full-correct-count target on declared 30-checkpoint classes. At the same six-item budget, native Fluid leaves 18/57 cross-count collision pairs on ARC/GSM8K; the class-fitted supports leave 0/0. Table 22 reports collision and ambiguous-unit counts; Figure 10 gives budget curves through 100 items. Transcripts include both item IDs and binary feedback. Static/greedy discovery uses the full declared class, whereas Fluid uses the released IRT bank, so this retrospective comparison does not establish equal-calibration prospective superiority. This supplies one native recent-method baseline; Appendix M's adaptation is not a native ATLAS or tinyBenchmarks reproduction. Pinned-source replay checks and historical-output deviations are disclosed in Appendix O.3.

  1. Model coverage and independent task text (W2). Matched Gemma2-9B-IT, Mistral-7B-Instruct-v0.3 and non-thinking Qwen3-8B add 11,664 responses to the original Qwen2.5/Llama3.1 panel, forming five variants from four families (Appendix K). Original-text ProofWriter supplies frozen derivation/held-out testing (Appendix L), followed by fresh-theory repair validation in Appendix N. Native replay adds ARC/GSM8K evidence in Appendix O.

  2. Complete statistics and integer consistency (W3/Q1). Appendix I, Tables 13–15 now supply the omitted results. For the historical four-family balanced extension, model-wise values are:

Gemma: n_B=800, P(T|B)=0.918, P(Q|B)=0.960, difference 0.043 [95% CI 0.010, 0.077].

Llama: 452, 0.420, 0.867, difference 0.447 [0.358, 0.536].

Mistral: 528, 0.606, 0.841, difference 0.235 [0.119, 0.346].

Qwen: 620, 0.735, 0.868, difference 0.132 [0.085, 0.181].

Differences use unrounded counts. All four observed contrasts are positive, with pointwise source-cluster intervals above zero. This historical extension is unmatched; Appendix K, Table 16 gives separate expanded matched results. The second-source joint value is corrected to 216/444=0.486; asymmetry is 204/444=0.459. Main asymmetry intervals are unchanged; the secondary gap bootstrap now resamples complete clusters. Table 15 gives all legacy permutation fractions, including matched-balanced pooled 12/2,000=0.006. Their row shuffle does not preserve full cluster dependence, so these diagnostics are not calibrated p-values and do not establish independence or dependence.

We would appreciate your assessment of whether these same-endpoint budget curves, native-method results, and completed statistical reports resolve the comparison and reporting concerns.

Reviewer_UaNFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a model-evaluation protocol contains enough information to identify the behavioral property that its score is intended to measure. Rather than treating a high-precision score as evidence of measurement validity, the authors define protocol identifiability relative to a finite declared behavior class HH, an estimand τ\tau, and a set of available measurements MM. If two behaviors have different estimand values but identical observations under the selected measurement support, the protocol is under-identified. Such a cross-estimand collision provides an explicit witness of the failure. The authors then formulate the construction of a distinguishing measurement support, or the minimum augmentation of an existing protocol, as a hitting-set problem (Proposition 1; Algorithms 1–2). The framework is instantiated in controlled, solver-grounded reasoning tasks involving base, target-edit, and matched sham-edit worlds; label mappings; and alternative readouts. Exhaustive audits on seven hand-specified behavioral policies show that base-only evaluation leaves all policies observationally equivalent, whereas a synthesized two-cell support can identify the declared estimand. A nested synthetic stress class extends to 50 policies. The paper also analyzes matched target–sham responses from instruction-tuned models and, more substantively, derives a class of 86 behavioral signatures from model predictions. On this derived class, an exact 15-cell support resolves all cross-estimand pairs during derivation, leaves one collision pair on held-out re-audit, and permits a one-cell post-holdout repair. The reported findings are explicitly restricted to controlled testbeds. Strengths. The paper addresses an important distinction between the statistical precision of a score and the structural ability of an evaluation protocol to measure its intended target. Its audit-to-repair formulation is operational and the model-derived signature experiments, held-out re-audit, and adversarial class-enrichment analysis provide useful evidence beyond the initial synthetic policy examples. Weaknesses. The central characterization and optimization reduction are largely direct applications of classical identification and hitting-set machinery. A closely overlapping August 2026 public paper already reports the seven-policy audit, 36-cell protocol, leave-one-component-out witnesses, and two-cell optimum, making a precise accounting of the present submission's incremental contributions essential. Moreover, all empirical validations remain within controlled, solver-generated tasks; the framework's usefulness and measurement savings on independent natural benchmarks are not yet established, and some claimed model-wise statistics are not fully reported.

Critical strengths
  1. The paper turns a measurement-validity concern into an explicit audit-and-repair procedure. Sections 3–4 distinguish the ability to estimate a score precisely from the more fundamental question of whether the selected measurements identify the desired estimand. The audit returns a concrete pair of behaviors with different estimand values that remain observationally indistinguishable; this is more actionable than a generic warning that a benchmark may be incomplete. The corresponding distinguishing sets define minimum-support synthesis and minimum-augmentation repair as precise optimization tasks. Although these reductions use classical tools, their organization into a reproducible protocol-design workflow is useful for model evaluation.
  2. The experiments on model-derived signatures demonstrate a meaningful extension beyond hand-designed policies. Section 7.1 and Appendix G construct 86 unique full-panel signatures (62 distinct estimand values) from model outputs and freeze them before testing the selected support on the second source. The exact 15-cell support resolves all 3,626 cross-estimand pairs in the derivation class and leaves only one collision among 1,125 held-out cross-estimand pairs. By comparison, the same-budget entropy top-15 support leaves 72 unresolved derivation pairs and 31 ambiguous held-out units; the random baseline is also substantially worse in typical draws (Table 2 and Appendix G, Table 10). The residual collision is identified explicitly and repaired with an additional cell, although that step is properly labeled exploratory. This provides a concrete demonstration of the proposed workflow on observed model behaviors.
  3. The robustness and negative-result analyses help define the scope of the guarantees. The authors do not claim that an identifying support for one frozen behavior class will continue to identify every new alternative. Appendix H shows that adding unobserved mixture policies produces collisions under the originally certified 15-cell support, while re-auditing and augmentation can repair them at additional measurement cost. The nested H7H_7–H50H_{50} stress test, the complete 16-subset component lattice in Appendix C, and the matched target/sham controls also clarify what is established. In Section 7, the base-conditioned sham–target asymmetry is supported in the two controlled sources, while the joint selective-success rate is reported as close to its independence baseline. Reporting these boundaries makes the empirical analysis more credible.
Critical weaknesses
  1. The independent novelty is limited by substantial overlap with existing methods and a directly related earlier preprint. Proposition 1 is an elementary finite-class identifiability condition, while minimum measurement synthesis and repair reduce to conventional hitting-set/model-discrimination problems. More importantly, the publicly available August 2026 paper Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation (arXiv:2608.13326) already describes seven deterministic behavioral policies, a 36-cell measurement tensor, base-only observational collisions, leave-one-component-out witnesses, and a minimum two-cell identifying support. These are central elements of the present submission's early sections and structural experiments. The current work does add a model-derived 86-signature analysis, held-out repair, richer matching, and class-enrichment experiments, but its presentation does not clearly separate these new contributions from the already documented framework. This issue is important when evaluating the paper's claimed methodological novelty and impact; overlap alone should not be taken as an allegation about authorship or intent.
  2. The practical identification and efficiency claims remain confined to a highly controlled and partially retrospective setting. The structural guarantees hold only for a finite, declared HH, τ\tau, and candidate measurement family. For the more realistic model-derived setting, the 86-signature class is built using complete 54-cell response records before a smaller support is synthesized. This convincingly demonstrates retrospective redundancy detection and prospective auditing within a closely related held-out controlled source, but it does not establish that the resulting support can identify previously unobserved behaviors on a genuinely independent natural-language benchmark. Appendix H directly shows that unseen alternatives can invalidate a frozen certificate. Moreover, the post-holdout one-cell repair uses information from the holdout itself and should not be treated as a second independent validation. The paper would benefit from an external benchmark or a prospective evaluation demonstrating actual measurement-cost savings and the rate of newly discovered collisions.
  3. Some headline descriptions and supporting empirical reports do not fully align with the precise estimand and available evidence. The statement that the 15-cell support identifies all 86 derivation signatures conflates identification of a finite-valued estimand with recovery of each complete behavioral signature. Since the 86 signatures have only 62 distinct estimand values, the framework does not generally require separating same-estimand behaviors. In addition, Section 7 states that the positive matched target–sham asymmetry holds across four model families and that per-model estimates with confidence intervals appear in Appendix I. The supplied Appendix I reports aggregate readout results and human-audit integrity, but not a transparent four-model effect table. The pooled matched-control findings are useful, yet the cross-family conclusion deserves directly inspectable per-model denominators, effects, and uncertainty intervals. Greater precision in these claims would strengthen the paper's credibility.
Questions for authors
  1. Regarding novelty and the relationship to directly preceding work. Could the authors explicitly compare the present submission with Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation (arXiv:2608.13326, posted August 13, 2026)? Which definitions, policy classes, measurement cells, collision results, and synthesized supports were already available, and which theorems, algorithms, and experiments are new in this submission? In particular, what is the distinct methodological contribution beyond applying classical hitting-set machinery to cross-estimand collisions?
  2. Regarding external validity and practical measurement savings. Can the authors demonstrate the frozen audit-to-repair workflow on an independent, natural-language evaluation task or on genuinely new model behaviors that were not used to define the initial measurement class? Since the model-derived class was constructed using full 54-cell signatures, how should a practitioner measure the net evaluation cost, including the up-front discovery and re-audit costs? What fraction of new behaviors causes collisions after synthesis, and how often would successive repairs be required in a prospective setting?
  3. Regarding the precise identification claim and the matched-control evidence. Does the frozen 15-cell support guarantee recovery only of the estimand over the 86 derivation signatures, or recovery of each full behavioral signature? Could the abstract and Section 7.1 distinguish these notions explicitly? Also, could the authors provide the four model-wise values of P(T∣B)P(T\mid B) and P(Q∣B)P(Q\mid B), their common base-correct denominators, target–sham asymmetries, and confidence intervals referred to in Section 7 and Appendix I? A transparent table would help assess whether the pooled effect reflects a consistent cross-family phenomenon.
1 comment
AuthorsOpenReview ↗

Response to UaNF: contribution, frozen repair transfer, and reporting

Thank you for the detailed review. The revision clarifies the contribution, adds a repair validation frozen before new observations, and provides the missing model-wise statistics.

Locations: Section 7.2; Appendix N, Tables 19–20 (p. 25); Appendix I, Tables 13–15 (p. 20); Appendix K, Table 16 (p. 22).

  1. Preprint, classical methods, and contribution (W1/Q1). arXiv:2608.13326 is an earlier version of this study. Its finite criterion, H7 policies, 36-cell tensor, component witnesses, and two-cell optimum are shared results, not new results claimed here. The original submission added H7–H50 stress classes, the model-derived bridge, class enrichment, and matched experiments. Section 2.3 and Appendix B explicitly credit decision-relative reducts/minimum-feature selection. Proposition 1 and Algorithms 1–2 assemble known identification and hitting-set tools; we do not claim a new theorem or hitting-set algorithm. The contribution is an executable evaluation workflow that turns an under-identified inference into an inspectable collision witness and a support-preserving repair, evaluated on actual model responses.

  2. Frozen repair, external evidence, and costs (W2/Q2). Section 7.2 and Appendix N report a repair fixed before 96 new original-text ProofWriter theories, disjoint by theory ID and content hash from all 84 earlier theories. Two models yield 192 model–theory units. Original 12-cell lookup recovers the complete four-count target on 45/192; repaired 17-cell lookup recovers 72/192, a paired gain of 14.06 percentage points [95% theory-cluster bootstrap interval 9.38, 18.75]. Covered wrong predictions fall from 37 to 7; 113 units still abstain (Table 19). The selected-cell predictions were saved before the remaining full-reference observations.

The 45/192-to-72/192 contrast compares two frozen pipelines, changing the behavior class/lookup as well as support; it is not a pure selector effect. Holding the enriched class/lookup fixed, adding five measurements raises recovery from 34/192 to 72/192. At equal 17-cell budget, exact, greedy, conditional entropy, and target information gain each recover 72/192, while 32 random supports average 50.09/192 (range 35–68; Table 20). The selector comparison is a secondary retrospective replay, not an additional independent validation.

A full-reference re-audit of the enriched class plus this batch finds 113/192 ambiguous validation units at 12 cells versus 21/192 at 17; these are retrospective per-unit ambiguity counts, not prospective collision-arrival rates. The earlier one-cell post-holdout repair remains exploratory. A practitioner would first acquire discovery panels, freeze class/support/lookup, then query selected cells on new units; unseen or ambiguous projections abstain, with periodic full-reference re-audits before any further repair. Accounting includes 7,776 earlier and 10,368 new responses (3,264 selected plus 7,104 remaining references). Neither net deployment savings nor the frequency of repeated future repairs is established. Transfer is limited: the original bridge lookup recovers 3/48 held-out units, and a new-family test recovers 0/216 (five covered).

  1. Precise target and complete reporting (W3/Q3). The guarantee identifies the estimand over 86 derivation signatures with 62 target values; it does not generally require separating same-target signatures. In this instance the 15-cell projection also distinguishes all 86. The abstract and Section 7.1 now make this distinction. Appendix I, Tables 13–15 provide historical model-wise denominators, target/sham estimates, intervals, and legacy permutation diagnostics; Appendix K, Table 16 adds the expanded matched panel. The original four-family extension is labeled unmatched. Table 1's second-source values are corrected to joint 216/444 = 0.486 and sham-minus-target 204/444 = 0.459.

These additions validate a frozen repair beyond its design observations while retaining the declared-class, task, and cost limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision summary: frozen repair, benchmark consequence, and complete statistics

Thank you for the constructive reviews. We have revised the manuscript to address the main concerns about novelty, independent validation, practical relevance, baseline comparisons, and statistical reporting.

The central question remains unchanged: do the observed measurements identify a declared target, and which additional measurements repair a failed audit?

Key additions in the revised PDF:

  1. Frozen repair on new observations: On 96 new ProofWriter theories (192 model–theory units), complete-target recovery improves from 45/192 to 72/192, a paired gain of 14.06 percentage points [95% CI 9.38, 18.75]. See Section 7.2 and Table 19 (p. 25).

  2. Equal-budget selection comparison: Exact, greedy, and effective information-based heuristics tie at 72/192 under the same 17-cell budget. This secondary retrospective comparison is reported in Table 20 (p. 25).

  3. Real-benchmark consequence: In released ARC-Challenge data, two checkpoints have identical responses on 12 native Fluid-selected questions but differ by 9.73 percentage points in full accuracy. Appendix O reports this collision, native Fluid results, and budget curves (pp. 26–27).

  4. Complete statistical reporting: Appendix I, Tables 13–15 provide the historical model-wise statistics and uncertainty intervals. Appendix K, Table 16 reports expanded matched-model results.

  5. Answer persistence: Appendix P, Table 23 (p. 28) decomposes the observed target–sham contrast and explicitly states that a persistence-adjusted causal effect is not identified.

Section 2.3 and Appendix B also clarify the correspondence to classical decision-relative reducts and hitting-set methods. We do not claim a new separation theorem.

The revised manuscript preserves the original research question and remains within the 10-page main-text limit. We have provided reviewer-specific replies with detailed evidence and locations.

We thank the reviewers for the suggestions that motivated these improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.