- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies framework routing. The idea is that some questions must be answered inside one epistemic framework before any solving begins. A coin flip question asked as a null hypothesis test needs frequentist statistics. The same scenario asked with a prior needs Bayesian statistics. Mixing the two is not a partial answer. It is incoherent. The authors build a benchmark with twelve such categories, train a small router to pick the right one, and then audit their own work.
The audit is the real content. They find that a classifier that cannot read the words, only punctuation and shape, nearly matches the real router. They find that most ill-posed training items carried a giveaway parenthetical that announced the label, present in training but nearly absent at test. They release a repaired version that removes the marker, and they keep the old version intact so prior numbers still reproduce.
The repair fixes the benchmark but not the capability. After removing the marker, benchmark recall on ill-posed items jumps. But on a certified set, where labels are computed from the generating parameters rather than annotated, the router detects none of the ill-posed items, with or without the marker. A pre-registered replication on eight other generator families repeats this. They also find no measurable effect of gate hardness at matched capacity, and a small specialist that beats the matched gates but differs from them in several ways at once, so that comparison is not controlled.
Critical strengths
1/The honesty is the real contribution, and it is exceptional. The paper's main result is that its own benchmark is largely solvable from surface form, and that its own router learned nothing about ill-posedness. It reports this plainly, versions the repair, keeps the flawed version for reproducibility, and discloses the leakage the repair does not fix. 2/The certified twins are a strong instrument. Labels come from the generating parameters and are rechecked by independent code that does not import the generator. This separates recovering a keyword rule from recognizing the epistemic property, which annotation based benchmarks cannot do. 3/The claims are carefully bounded. No detected gate effect is stated as a failure to detect, not an equivalence. The specialist margin is called observational. The propositions are offered as criteria, not as results any routing number verifies. 4/The statistical discipline is high. The paper uses paired bootstraps, pre-registration, cluster bootstraps for near duplicate families, and an explicit note that finite suite counts are not binomial ceilings. 5/On novelty: routing by epistemic compatibility, rather than by cost or quality, is a genuine reframing of the routing literature. The certified twin method is a real methodological contribution on its own, separate from whether the router works. The closest prior work is Molavi et al. on hard routed reasoning LoRAs, which the paper reimplements correctly as its hard gate.
Critical weaknesses
1/The paper demonstrates no working capability. This is the deciding issue. After the audit, the router detects none of the ill-posed items on either certified set, with or without the marker. There is no detected gate effect. The one downstream test shows that conditioning the solver lowers answer accuracy. So the paper offers no evidence that framework routing is learnable with this approach, and it does not claim otherwise. For a paper whose stated goal is a routing method, a clean null on the authors' own instrument is not enough to accept. 2/The theory never touches the experiments. The two propositions concern the output space, but every routing number measures a twelve way classifier that the paper itself shows can be driven by surface form. Nothing in the evaluation tests the propositions. They motivate the problem but do no work, so the formal content does not strengthen the empirical case. 3/The central taxonomy is unvalidated. The twelve labels are rule based, admitted to be operational rather than true universes, and the rules are shown elsewhere in the paper to be leaky. The one human audit that would establish construct validity is unfinished, and its pre-registration order was broken when an AI assistant opened the answer key early. The object the whole paper rests on is therefore not yet established.
Questions for authors
1/Is framework routing learnable at all with this approach? A single positive result, even on one framework pair, would change the recommendation. As it stands the certified sets show nothing is learned. 2/Can the propositions be tied to a measurement? Is there any diagnostic where the interior mixture penalty is shown to bite, rather than argued? 3/The framing needs a validated regime test, which the paper says it lacks. Without it, how does a practitioner know when hard commitment is required rather than harmful? 4/Can a clean independent human validation of the label boundary be completed before publication, given that the broken order voids the current one? 5/Does framework routing ever help end to end? The one downstream result points the other way.