acceptodds
Withdrawn from ICLR 2027

When Routing Accuracy Misleads: Auditing Framework Selection with Certified Twins

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many reasoning tasks require committing to a single epistemic framework before solving: blending a -value with a posterior yields incoherence, not a partial truth. We study framework routing — selecting the applicable belief-space "universe" before reasoning — and report that high routing accuracy is a poor measure of it. Auditing our own benchmark, we find that a classifier that cannot read the words, only their shape and punctuation, reaches % against a % majority baseline, and that most Ill-posed training items announced their own deficiency in a parenthetical present in % of train but % of test items. We release EpistemicBench v5, an explicitly versioned repair — v4 is untouched — that removes that marker together with the train-only augmentation tags, and discloses the leakage it does not fix. The repair works and does not buy the capability: retraining lifts ill-posed recall by pp, yet on a certified subset whose labels are machine-checkable properties of the generating parameters, our router detects ill-posed items, trained with the marker or without it. A pre-registered, unaudited replication over eight disjoint generator families repeats this at . Items within a family are near-duplicates, so these are finite-suite counts, not binomial ceilings. Benchmark accuracy on this class can therefore be a severe overestimate of epistemic competence, and we make no claim that ill-posedness detection was learned. On the repaired data we detect no effect of gate hardness: every paired 95% CI among matched M hard, soft and Bayesian gates includes zero — a failure to detect a difference, not a demonstrated equivalence. A M fine-tuned specialist beats all three and ties RoBERTa-large at its trainable budget, but it differs from them in architecture and objective at once, so that comparison is observational. EpistemicBench v5, both certified sets, the audit tooling and code will be released upon publication. Audit the benchmark before believing the number.

Primary area:
datasets and benchmarks
Keywords:
benchmark auditing, shortcut learning, dataset leakage, epistemic frameworks, framework routing, certified evaluation, minimal pairs, EpistemicBench

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_mphDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies framework routing. The idea is that some questions must be answered inside one epistemic framework before any solving begins. A coin flip question asked as a null hypothesis test needs frequentist statistics. The same scenario asked with a prior needs Bayesian statistics. Mixing the two is not a partial answer. It is incoherent. The authors build a benchmark with twelve such categories, train a small router to pick the right one, and then audit their own work.

The audit is the real content. They find that a classifier that cannot read the words, only punctuation and shape, nearly matches the real router. They find that most ill-posed training items carried a giveaway parenthetical that announced the label, present in training but nearly absent at test. They release a repaired version that removes the marker, and they keep the old version intact so prior numbers still reproduce.

The repair fixes the benchmark but not the capability. After removing the marker, benchmark recall on ill-posed items jumps. But on a certified set, where labels are computed from the generating parameters rather than annotated, the router detects none of the ill-posed items, with or without the marker. A pre-registered replication on eight other generator families repeats this. They also find no measurable effect of gate hardness at matched capacity, and a small specialist that beats the matched gates but differs from them in several ways at once, so that comparison is not controlled.

Critical strengths

1/The honesty is the real contribution, and it is exceptional. The paper's main result is that its own benchmark is largely solvable from surface form, and that its own router learned nothing about ill-posedness. It reports this plainly, versions the repair, keeps the flawed version for reproducibility, and discloses the leakage the repair does not fix. 2/The certified twins are a strong instrument. Labels come from the generating parameters and are rechecked by independent code that does not import the generator. This separates recovering a keyword rule from recognizing the epistemic property, which annotation based benchmarks cannot do. 3/The claims are carefully bounded. No detected gate effect is stated as a failure to detect, not an equivalence. The specialist margin is called observational. The propositions are offered as criteria, not as results any routing number verifies. 4/The statistical discipline is high. The paper uses paired bootstraps, pre-registration, cluster bootstraps for near duplicate families, and an explicit note that finite suite counts are not binomial ceilings. 5/On novelty: routing by epistemic compatibility, rather than by cost or quality, is a genuine reframing of the routing literature. The certified twin method is a real methodological contribution on its own, separate from whether the router works. The closest prior work is Molavi et al. on hard routed reasoning LoRAs, which the paper reimplements correctly as its hard gate.

Critical weaknesses

1/The paper demonstrates no working capability. This is the deciding issue. After the audit, the router detects none of the ill-posed items on either certified set, with or without the marker. There is no detected gate effect. The one downstream test shows that conditioning the solver lowers answer accuracy. So the paper offers no evidence that framework routing is learnable with this approach, and it does not claim otherwise. For a paper whose stated goal is a routing method, a clean null on the authors' own instrument is not enough to accept. 2/The theory never touches the experiments. The two propositions concern the output space, but every routing number measures a twelve way classifier that the paper itself shows can be driven by surface form. Nothing in the evaluation tests the propositions. They motivate the problem but do no work, so the formal content does not strengthen the empirical case. 3/The central taxonomy is unvalidated. The twelve labels are rule based, admitted to be operational rather than true universes, and the rules are shown elsewhere in the paper to be leaky. The one human audit that would establish construct validity is unfinished, and its pre-registration order was broken when an AI assistant opened the answer key early. The object the whole paper rests on is therefore not yet established.

Questions for authors

1/Is framework routing learnable at all with this approach? A single positive result, even on one framework pair, would change the recommendation. As it stands the certified sets show nothing is learned. 2/Can the propositions be tied to a measurement? Is there any diagnostic where the interior mixture penalty is shown to bite, rather than argued? 3/The framing needs a validated regime test, which the paper says it lacks. Without it, how does a practitioner know when hard commitment is required rather than harmful? 4/Can a clean independent human validation of the label boundary be completed before publication, given that the broken order voids the current one? 5/Does framework routing ever help end to end? The one downstream result points the other way.

Reviewer_WD6BOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes framework routing: deciding which reasoning framework a question belongs to (frequentist vs. Bayesian, classical vs. quantum, Euclidean vs. non-Euclidean, and so on; twelve labels in total) before trying to solve it. They build a 22k-sample benchmark, named EpistemicBench, labeled mostly by keyword rules over existing corpora, and train a small router to do the classification.

Most of the paper is about auditing that benchmark. The authors find that a classifier which sees only punctuation and word shape gets 76%, that the source corpus alone predicts most labels, and that 90% of ill-posed training items contain a parenthetical that literally announces the label. They remove the parenthetical, which raises ill-posed recall by ~48pp on the benchmark. But on a small set of “certified” minimal pairs, the router detects 0 of 30 ill-posed items, with or without the fix, and 0 of 100 on a larger pre-registered set. The paper also compares capacity-matched hard, soft, and Bayesian gates, finding no statistically detected difference.

Critical strengths
  1. The artifact ablation is a strong part of the paper. Removing the deficiency markers and augmentation tags leads to substantial improvements in ill-posed recall for both Qwen and RoBERTa. However, the router still fails to identify any of the 30 ill-posed questions in the certified pairs, whether trained with or without these markers. This clearly shows that better benchmark scores do not necessarily reflect better recognition of ill-posed questions.
  2. The certified minimal pairs are also well designed. Each pair differs by only one clause: one question is missing information needed for a unique answer, while the other provides it. Rather than relying on annotated labels, the authors verify ill-posedness by showing that two valid values of the missing quantity lead to different answers, with independent code checking the results. Since the questions are otherwise very similar, this provides a more meaningful test of whether the model can recognize when important information is missing, rather than simply relying on keywords.
Critical weaknesses
  1. The paper lacks a clear focus and a coherent presentation. Although the benchmark audit is an interesting contribution, much of the paper is devoted to loosely related topics, including MoE routing, theoretical propositions, continual learning, uncertainty estimation, and gradient analysis. It is often unclear how these parts contribute to the main research question. The paper also spends too much space discussing limitations, mistakes in earlier drafts, and subsequent corrections. Many of these details would be better suited for an appendix or changelog rather than the main paper.
  2. Insufficient independent validation raises concerns about the reliability of the findings. Given the extensive use of AI tools disclosed by the authors, I think independent human validation is particularly important. However, the benchmark still lacks a completed independent human audit of its ambiguous labels, and the planned audit did not follow its pre-registered protocol because an AI assistant accessed the answer key. The certified evaluation is useful, but the routing experiments mainly focus on the authors' own model and lexical baselines. As a result, it remains unclear how well the findings generalize beyond the specific models and evaluation settings considered in this paper.
Questions for authors
  1. Have RoBERTa-large or the matched MoE gates been evaluated on the certified twins or the stress set? If they also fail, the finding is more general.
  2. Has anyone other than the authors checked that the rendered certified questions state exactly the constraints their certificates assume? Is there any human-validated subset of the benchmark labels?
  3. Do the training data and certified tests use the same definition of "ill-posed"? In particular, how do you distinguish questions that can be answered with additional assumptions from those that should be labeled ill-posed?
Reviewer_rj5cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates LLM routing based on epistemic frameworks. Intuitively, an (ideally lightweight) classifier would analyze an incoming question qq and decide to send it to a corresponding model skilled in one of twelve selected reasoning frameworks: frequentist vs Bayesian statistics, classical vs quantum physics, etc. In practice, in this paper, there is a single expert LLM model: the routes differ simply in the prompt used, which guides the LLM to use a corresponding epistemic framework. The motivation is that querying LLM⁡(q∣fwk⁡(q))\operatorname{LLM}(q|\operatorname{fwk}(q) ) would maybe be more accurate than directly querying LLM⁡(q)\operatorname{LLM}(q). However, the paper results in Section 4.5 find that this is not the case, and conditioning on the framework actually hurts predictive performance.

Still, the paper proposes EpistemicBench, a benchmark for classifying a question into the above-mentioned twelve frameworks. The benchmark is constructed with questions from existing datasets such as MMLU, MATH or SciQ. The framework labels are assigned by the authors using domain-specific keyword rules. The paper mentions the proposed version (v5) and an earlier version (v4), to highlight incremental improvement and fixes they have introduced (not sure the numbering matters) .

Then, the paper evaluates different classification baseline which are essentially LLM encoders + a classification head, each fine-tuned on the task (either full fine-tuning or LoRA). The model proposed by the authors, a fine-tuned QWEN (which is said to be "specialized", though I am unsure what is its architectural novelty), performs comparatively to the other models using less parameters.

But, here comes the main finding of the paper: even when a model is very accurate on the proposed benchmark, it has poor accuracy on a carefully crafted set of hard examples (the "twins"). This shows that the constructed benchmark is, in fact, a poor proxy for the capability it intends to evaluate. The paper argues that this shows that epistemic understanding is therefore difficult to evaluate, and that this insight is its main contribution.

Critical strengths

The authors acknowledge a lot of the limitations of their work in the manuscript. In particular, that the routing motivation (conditioning an LLM on the framework) is not empirically verified and that the benchmark is a poor proxy for epistemic understanding. In that, the paper is honest. Still, even if these limitations are acknowledged, I believe they are significant.

Critical weaknesses

The first critical weakness is in the motivation of the paper. As mentioned in the summary, the premise of the paper is that routing for epistemic competency is useful. But, empirically, the paper shows that this is not the case with their proposed routing strategy: the LLM with a prompt augmented with framework-hints performs worse on the question-answering tasks than an unconditioned LLM. While this is acknowledged by the authors, there is still nothing to advocate for epistemic framework classification being an important task deserving a dedicated evaluation. In other words, it is unclear why the work is significant for the ML community.

Secondly, ignoring the empirical results, the investigation of "epistemic reasoning" is here superficial. The formalization of belief-space universes and solvers relying on them is orthogonal to the approach used in the paper: these "belief-spaces" are used as mere keywords injected in prompts, passed to LLM which are famously informal reasoners. I would have expected a deeper technical novelty, where an agent can use different symbolic solvers as tools to derive the answer, each with their own set of axioms (i.e., something neurosymbolic).

Finally, I think the conclusion that epistemic competence is generally difficult to evaluate, which the paper advocates is its strongest contribution, is hasty. Currently, the paper shows the accuracy on the constructed dataset does not translate on a carefully constructed set of examples. Indeed, the task could be difficult as advocated by the authors; OR, the dataset could simply not cover a wide enough distribution. These are two very different issues.

Questions for authors

Is there an architectural novelty in the QWEN + LoRA approach? What makes it more "specialized" than, let's say, RoBERTa? Aren't both fine-tuned on the same dataset?

Reviewer_2UyGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper constructs a new dataset called EpistemicBench which contains 22K samples spanning 12 universes across 5 domains. The universes are belief-space universes which represent non-overlapping axiomatic and reasoning systems. The original goal was to build a classifier to map a question into the belief space necessary to solve that question, plausibly based on the terms or phrases that hint at what kind of belief system is useful (e.g bayesianism vs frequentism). The paper uses LORA to finetune a Qwen 1.5 0.6B param classifier and finds that although the classifier fits to the data, accuracy on the benchmark doesn't not necessarily show

Critical strengths

One strength is that the area of investigation seems to be genuinely interesting. It will be useful to have a proper understanding of how and whether LLMs (or agentic systems that use LLMs) can accurately categorize or recognize what set of axiomatic principles to work with for a question that is ambiguous in the sense that multiple different reasoning frameworks can be applied but the question formulation indicates a preference for one. It will also be useful to have a dedicated benchmark or analyze specific issues around why coming with such a benchmark will be difficult. The discussion of belief space universes and what constitutes a belief space universe is also interesting.

Critical weaknesses

This paper's experimental set up is very weak. There is only one results table comparing the classification accuracy to non-state of the art methods. For example, the paper could have just use zero shot modern LLMs to perform classification on the final version of the benchmark being released, to see how it compares to the fine-tuning. The paper also proceeds with running these experiments after finding that the benchmark, even after repair does not measures "epistemic competence". A lot of the paper text reads like it was LLM generated especially section 4 and 5.

Questions for authors

How would you construct a benchmark that actually measures "epistemic competence"? Why were readily available frontier models not used at all as a classification baseline in the main results table?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.