acceptodds
Withdrawn from ICLR 2027

When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern learned systems increasingly combine learned components with search, repair, or external solvers. Benchmarks often measure the resulting end-to-end system, while scientific claims may concern only one component, creating an attribution problem: evidence can fail to support the requested component-level claim while still supporting a positive conclusion about the larger system. Existing evidence-to-claim methods primarily calibrate claim strength. We argue that composite systems require a second dimension: scientific subject. We address this problem with subject-typed claim licensing, which separates weaker conclusions about the requested subject from positive but non-substitutive credit about another subject. We instantiate this idea in SCOPE-ROUTING for preference-conditioned multigraph routing. Non-authors reproducibly apply the declared semantics; held-out review yields fewer reference-relative upward deviations than unstructured review, while the difference from a strong evidence checklist remains unresolved; and a controlled routing study shows that score-optimal and claim-eligible methods can differ while valid hybrid-system credit is preserved. These results motivate treating claim strength and scientific subject as distinct dimensions of evidence-based evaluation.

Primary area:
datasets and benchmarks
Keywords:
machine learning evaluation, benchmark validity, evidence-to-claim reasoning, scientific claim attribution, composite learning systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_weHEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the "scientific attribution problem" in modular AI systems: the final performance of such a system may emerge from the combined contributions of learning models, external solvers, and retrieval or refinement modules, yet researchers often attribute the system's overall performance solely to one of its learning components. To tackle this, the authors instantiate this concept through "SCOPE-ROUTING," a framework for preference-conditioned multigraph routing.

Critical strengths

S1. Scientific attribution is a genuine and meaningful problem.

S2. Separating "weaker same-subject claims" from "positive cross-subject credit" is a sound design for representation.

S3. The experimental design goes beyond mere theory or synthetic examples.

Critical weaknesses

1.The central contribution of cross-subject retained credit is not directly validated

Does the improvement in review accuracy reported in Table 2 actually validate the newly introduced "subject-typed representation," or does it merely reflect the impact of employing fixed evidence requirements and a "claim ladder"? The authors note in §6.2 that five of the eight "narrowed cases" still retain positive "lower-scope credit"; however, they do not provide a systematic human evaluation of accuracy, false-negative rates, and false-positive rates specifically for cross-subject credit.


2.No established improvement over the strongest baseline, despite increased complexity

There is currently no clear evidence that a fully formalized SCOPE-ROUTING is superior to a well-designed evidence checklist.


3.The evaluation relies heavily on author-defined claim contracts and contract-relative labels

The evaluation framework in this paper relies on a "C0–C6 claim ladder" devised by the authors, alongside pre-specified evidence obligations. For instance, a C5 claim must satisfy the requirements inherited from levels C0–C4, with the additional inclusion of evidence for dynamic robustness. While these rules are reasonable, they do not necessarily constitute the only valid scientific claim hierarchy.


4.The methodological presentation is unnecessarily abstract relative to the actual adjudication procedure

The text introduces a significant number of abstract symbols, eight-tuple "contracts," and routing-specific mathematical formalisms. While these formalisms are not entirely devoid of meaning, they have not been subjected to sufficiently rigorous theoretical analysis or empirical testing.

Questions for authors

same as weaknesses

Reviewer_NAJYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper suggests that in composite systems, an easy benchmark score may be an indicator of the entire system rather than just the learned part. The authors suggest subject-typed claim licensing that can be weak in the same subject (the learned policy) but to preserve the positive evidence about a different subject (the hybrid system) so it is recorded separately and can't replace the asked claim. This is realized as SCOPE-ROUTING, a C0-C6 claim ladder for learned multigraph routing.

Critical strengths
  1. Hybrid system end-to-end scores are assigned to the learned component and the paper retains the system-level credit rather than throw it away.

  2. The human studies are meticulously constructed, blinding, papers held out, and clustered intervals. The authors are forthright that the acquire over the test is no longer statistically resolved.

Critical weaknesses
  1. This is almost common-sense. Reviewers already "say which component a result is about" and almost anything else poset formalism and propositions add is almost trivial, since the propositions are almost trivial. The paper itself admits that power only frameworks can be made to work with typed conclusions, so the conceptual improvement is very limit.

  2. The routing result is what it is by construction. A policy that is used 86% of the time in the backup is not obviously some proof of a policy we have learned. The fact that the 15 policy families seem to be boundary cases for certain gates is perhaps not surprising.

  3. The cross subject case is very thin. Fallback is the only tangible test case of matter change, most of the retained credit is about source/task scope, which is nearer existing strength and scope view.

  4. The study is small and may bias the method. For the 36 claims that have been settled, the CI that encompasses zero.

  5. The domain is hard to change. The contract is hand-tuned and domain-specific. C0-C6 levels and gates are the authors' decision. Writing a contract for a new domain costs nothing, and there is only a rough notion of portability.

  6. Hard to read. The paper uses its own vocabulary, e.g., licensing, retained credit, R, etc.

Questions for authors
  1. Are there any instances in Claim Scope-120 and the held-out set where it is actual cross-subject credit, rather than same-subject narrowing or source scope? That is, it would be helpful to know whether the "subject" axis provides any value over simply calibrating using strength.

  2. A strength-based approach with the easiest possible "system level note" fails or produces a worse outcome than yours?

  3. Were the Route Scope-Lite policies meant to activate particular gates? If yes, what does the image of the gate selection reveal beyond an example that was fabricated?

  4. How long to draft a new domain contract? How consistent are independent experts on levels and gates?

Reviewer_MgJoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes subject-typed claim licensing to distinguish a weaker supported claim about a requested component from positive evidence about a different component or composite system. A declared contract specifies claim ordering, evidence obligations, source admissibility, and separately retained credit. SCOPE-ROUTING instantiates the framework for preference-conditioned multigraph routing. Evaluation includes independent reconstruction of 24 claim objects, four-expert annotation of 120 claims, a held-out comparison of three review conditions on 40 claims, and a semisynthetic routing study with 1,500 requests and 15 policy families. The held-out study finds higher reference agreement than unstructured review, while the advantage over a strong evidence checklist remains statistically unresolved. The routing example illustrates how selection changes when eligibility for a component-level claim is required.

Critical strengths
  1. A useful representation of attribution in composite systems. Separating requested-subject support from retained system-level credit addresses a practical reporting problem. The fallback-heavy routing example makes the distinction concrete: low end-to-end regret can coexist with inadequate evidence for the learned component. The formal operator preserves positive evidence without allowing it to answer a different scientific question.
  2. The human evaluation goes beyond agreement with author labels. The paper includes blinded non-author reconstruction, held-out papers, a strong checklist condition, and a separate senior-expert panel. The 19.4-point agreement improvement over careful review has a paper-clustered interval of [5.6, 33.3]. The authors appropriately report the checklist comparison as unresolved and acknowledge increased downward deviations, purposive sampling, and the normative nature of the contract. These qualifications substantially improve interpretability.
Critical weaknesses
  1. The evaluation does not isolate the main representational novelty. The reviewer comparison adds the declared claim hierarchy and maximal-selection rule to the checklist; the reported outcomes mainly measure agreement on supported scope. It does not separately quantify whether subject typing improves correct component attribution or retention of valid cross-subject credit. The routing study demonstrates a constructed attribution example, while the formal separation follows from the declared subject-preserving order and non-substitution rule. A matched comparison retaining the same evidence fields and hierarchy while varying the explicit subject and retained-credit channels would more directly test the paper's central contribution.
  2. Reference agreement remains partly dependent on the proposed ontology. The senior panel is blind to the ladder and produces free-form conclusions, which is a meaningful safeguard. However, Appendix E.5 says the study team subsequently maps those conclusions through the contract-resolution procedure. Agreement with that mapped reference therefore measures alignment with an ontology as well as scientific judgment. With 36 resolved claims and eight reviewers, the checklist difference of 11.1 points has an interval of [-2.8, 25.0]. Downward events also increase from three to four relative to the checklist. The paper acknowledges these trade-offs, but practical superiority and contract-independent validity remain open.
  3. The key routing decision is difficult to audit from the manuscript alone. Tables 16–18 provide regret, fallback, and hypervolume summaries, but list crucial route-change, latency, conditional-feasibility, and attribution checks as artifact-reported quantities or required checks. Table 17 states that cost normalization preserves the selected policy without presenting its numerical inputs. Since Shock-Safe Online Repair's C5 eligibility depends on those observations, the central example needs the actual measurements and decision thresholds in the paper. The artifact is described as containing these records; this is a reporting and auditability concern, rather than evidence that the analyses were never conducted.
Questions for authors
  1. Can you compare subject-typed licensing with an otherwise identical hierarchy-based procedure that omits the separate subject and retained-credit channels?
  2. How often do reviewers correctly identify the supported scientific subject and preserve valid cross-subject credit? Please report these outcomes separately from claim-level agreement.
  3. Can independent assessors evaluate the free-form conclusions before contract mapping, and can you report mapping disagreements and sensitivity to alternative reasonable contracts?
  4. Can the paper include the primary-only/end-to-end decomposition, exact conditional invalid-route counts, dynamic-response results, latency, and normalization used to establish C5 eligibility and selection?
  5. Can you report reviewer- and order-sensitive analyses, and clarify the aggregation of upward/downward events? Appendix E.5 explains that these can overlap with resolved agreement, so Table 2 would benefit from stating that convention explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.