W1. The "state-of-the-art" claim is refuted by the paper's own Table 3, and the baseline suite is both
dominated by a trivial baseline and applied outside the regimes its sources evaluate.
(a) The claim. The abstract states "Compared with the state-of-the-art methods, PARA improves average
score by up to 15.02 points", and Section 4.2 defines that comparison as HybridLLM, Router-R1 and Minions.
Those are not the strongest rows in Table 3: PARA (pool) averages 80.99, while GPT-5 Direct averages 85.66
(22.69 s, Ca 43.45) and GPT-5 ReAct averages 85.73. Recomputing per benchmark from Table 3, PARA is below
GPT-5 Direct on six of seven benchmarks — HotpotQA −7.05, GSM8K −2.25, DROP −4.75, HumanEval −7.31, MATH
−9.52, ToolBench −5.81 — and wins only FRAMES (+4.00); it is below All-LLM on all seven. The abstract's
"up to 15.02 points" is therefore a maximum taken over the three weakest comparators while the two
strongest sit in the same table.
(b) The baselines are not running where their source papers run. Every routing baseline, including the
"matched" variants that "share the same 16K SFT requests, GRPO rollout budget, executor pool, and
validation-based operating-point selection" (Section 4.1), scores below a single GPT-5 call: Router-R1
(matched) 77.88 and HybridLLM (matched) 73.50 are 7.78 and 12.16 points below GPT-5 Direct's 85.66 on the
seven-benchmark mean, and on HotpotQA the pair is 74.09 vs 83.97. In
Router-R1's own report the router beats the largest single model in the pool (+7.8 average EM points;
0.449 vs 0.416 F1 on HotpotQA), so the relative ordering is inverted here [public source: Router-R1,
arXiv:2506.09033, Tables 2–3]. I note that the absolute magnitudes are not comparable across the two
settings — Router-R1's pool tops out at Mixtral-8x22B and its main runs use cost coefficient α = 0.0 with
at most 4 routing rounds — so this may be a regime mismatch rather than a broken re-implementation. It is
nevertheless the authors' burden to report the configuration and show the inversion is not an artifact.
The same problem affects Minions: its published evaluation is long-document QA (FinanceBench, LongHealth,
QASPER) at 5.7×–30.4× cloud-cost reduction [public source: Minions, arXiv:2502.15964], whereas Table 3
reports Minions on HumanEval, MATH, ToolBench and FRAMES, with no description anywhere of how the
chunk-decomposition protocol was adapted to code generation or tool use. And the closest structural prior
work is absent altogether: "DoT (Shao et al., 2025) is closest structurally" (Section 5), yet DoT appears
in neither Table 3, Table 4 nor Table 5.
(c) No cost-matched control, and the omitted method's published operating point is at least as good. At
PARA's budget (Ca 31.50, 20.59 s) the natural control is one strong model with only a fraction of requests
going to the cloud, or with self-consistency; this is not run. Meanwhile DoT — which also builds a
dependency graph, executes same-depth subtasks in parallel, and routes between a local SLM and a cloud LLM
— publicly reports an 83.57% cloud-cost and 66.12% latency reduction at "comparable reasoning accuracy"
[public source: DoT, arXiv:2502.04392, §5.2 / Table 1], against PARA's 68.50% and 43.15% at a 3.85-point
accuracy loss. The settings differ (GPT-4o and Llama-3-8B vs GPT-5 and a four-SLM pool), so this is not a
head-to-head result and I do not treat it as one; but on the one overlapping benchmark where both report a
delta versus cloud-only, the sign differs — DoT reports DROP +5.59 points, while I recomputed PARA's
DROP gap from Table 3 as −2.68 (91.72 → 89.04). This is consequential because it is the difference
between "PARA advances the state of the art" and "PARA occupies a point on a Pareto curve that a one-line
baseline already dominates on quality and that a cited, structurally identical method may already dominate
on cost", and the abstract currently claims the former while declining to test either.
W2. Neither of the two mechanisms the paper is named for is instantiated, and one is contradicted by the
paper's own ablation.
(a) Error propagation is never operationalised. The router observes "the request, DAG topology ...,
current frontier, summaries of completed outputs, model executor profiles φ(m) and token prices"
(Section 3.2); nothing in that state is an error or propagation estimate, and the reward
R = r_succ − α Σ_t ΔCost_t (Eq. 2) has no propagation term. The ME / RB / RC / IRC quantities computed
in Section 2.3 never reappear at routing time, in training, or in any table of Section 4; the entire link
is one sentence in Section 2.4 ("Insight 2 motivates a router that accounts for both the current node and
upstream execution state"), which amounts to "the policy can read cached upstream text". Correspondingly,
Table 5 contains no condition that removes propagation information — no request-plus-topology-only state,
no per-node IRC estimate — so the title claim has no test.
(b) The adaptive horizon is a cost dial, not a propagation mechanism. Section 2.3 argues the horizon
should "shorten commitments before irrecoverable errors propagate", but Table 5(a) shows quality
decreasing monotonically in horizon length: Fixed h = 1 scores 81.84, Adaptive horizon 80.99, Fixed
h = ∞ 66.26. Section 4.4 reports this honestly ("with scores of 80.99 and 81.84, respectively") but never
reconciles it, and there is no analysis of when h_t is chosen or whether it correlates with dependency
depth or IRC risk.
(c) The replanning confound is not isolated. h = ∞ collapsing to 66.26 (−18.6 vs All-LLM's 84.84) shows
that boundary re-planning from cached outputs, not routing, does most of the work; but no ablation holds
re-planning fixed while routing everything to the cloud, so the 1.71–6.75 point gap to All-LLM conflates
routing gains with re-planning gains. This is consequential because the paper's identity — the title, the
abstract's three-part "jointly considers" claim, and the measured contamination argument in Section 2 —
rests on these two mechanisms, and as written a reader cannot tell whether PARA is a propagation-aware
router or a re-planner with a cost-aware model picker attached.
(Deliberately not listed as critical, but relevant to the recommendation: no score anywhere has a
confidence interval, seed count or significance test, and the evaluation-set size per benchmark is never
stated; Table 5(d) reports GRPO−SFT deltas to two decimals from what appears to be a single A100 run;
Appendix B.1's "350 request pairs" is "50 requests per benchmark and five repeats per request", i.e.
pseudo-replication, so its "[0.001, 0.253] s" interval is understated by roughly √5; Table 8 reports a
constant score of 82 while Ca varies 30.21 / 29.38 / 31.15 across cloud-concurrency caps, which should not
change total tokens, and gives latencies as "T(c) = 11.85 + 9.60/c" rather than measurements; Table 6's
caption calls its numbers "whole-benchmark scores" while Section B.1 refers to "50 requests per
benchmark"; and no code, checkpoint, planner prompt or data release is promised.)