The novelty and added value of the agent layer are not yet isolated from established record-based routing and tool-assisted configuration selection. Table 1 and §4.1 already provide nearest-neighbor recommendations, accuracy–cost frontiers, and held-out policy backtests. Section 4.2 specifies four recipe families, including “choose a policy,” in which the agent compares predefined policies but cannot invent them. Record-based routing that consumes new model evidence is also established in cited work such as UniRoute. Using an existing LLM to interpret this evidence may be useful engineering, but the paper does not show how much it adds beyond directly executing a selected statistical policy. An agent-free execution of the best backtested policy under the same evidence mask, objective, and preview budget—on both the 18- and 28-configuration DeepSWE pools—would clarify the contribution. Pool expansion in §5.6 already fixes the prompt and policy; the missing test is attribution to the agent rather than the underlying evidence and preset rules. Distinguishing the agent's own contribution from the benefit of a stronger pool is essential to the novelty claim.
The reported results do not yet establish a favorable end-to-end production tradeoff or robust transfer. Recipe design and selection are setup activities, while §4 says the LLM agent gathers evidence and submits a configuration for each routed query; §4.1 reports two or three tool calls per task after precomputation. The plotted USD 139.57 versus USD 180.22 LLMRouterBench model-call comparison leaves a USD 40.65 margin, but §5.2, §5.7.2, and §6 exclude the online agent's own inference bill. The larger DeepSWE USD 215 versus USD 500 model-call gap may be more forgiving, but also needs all-in measurement. Corpus reads count as free model calls, and previews are already charged; the missing quantities are point-level agent inference, tool/runtime overhead, latency, previews, and unresolved/retry rates, with offline corpus and recipe-development costs reported separately. Rejected previews should remain charged and accepted cached answers counted once, as the stated protocol requires. The three lowest-cost Fig. 2 points are already disclosed as +0.5-point-adjusted pilots with scaled costs (§5.2), and RLCascadeRouter is an identified imported published point (§5.2.1); unadjusted, cohort-labeled results, uncertainty, and matched split, pool, metric, and price conditions would make the frontier easier to assess. This is a comparability question, not a fabrication allegation. Exact prompts, tool schemas, and recipe selection matter because §4.2 reports 5.5–9.5-point sensitivity to recipe wording. The cross-benchmark result in §5.4 is a useful transfer test, but its paired advantage over the best single model includes zero; §5.6 adds models using records on the same DeepSWE tasks. Given the paper's own coverage and distribution caveats (§6), performance under sparse, stale, or shifting production evidence remains insufficiently demonstrated.
The stated evidence masks and fold-specific table rebuilding are useful safeguards, but reproducibility still requires a point-level manifest and an example preview payload showing what the agent can see and confirming hidden grading metadata is excluded from traces. Corpus-only settings average a configuration's recorded trials, whereas cascades score an accepted attempt (§4.4); each plotted point should identify its recipe and estimand. The existing task-bootstrap intervals are useful, while paired headline differences and routing repeats would characterize uncertainty from agent behavior. Finally, the completion-first objective's per-input success target is not shown to be implemented by the alpha-based, uncalibrated recipes. Reporting its heuristic status, coverage, and budget-exhausted cases would make this claim more precise. The retrospective behavioral-neighbor bound in Appendix A should remain distinct from a deployable retrieval result.
Thank you for highlighting the flexibility of COMPASS and the value of adaptive tool use. We address your two concerns below.
Response. The contribution of COMPASS is how the agentic layer uses evidence to make a routing decision for each task. Appendices D–F show three end-to-end routing trajectories where the agent overrides the fixed kNN recommendation. In the beta spectrometer example, the agent finds that the neighbouring tasks provide weak evidence and rejects both previewed answers. It then checks broader historical scores and selects a third model. These examples show how the agentic layer can assess the quality of the neighbours, choose another tool when needed, and revise the initial recommendation.
Response. Figure 2 now includes the router inference bill for each COMPASS point. In the headline LLMRouterBench run, DeepSeek-V4.1-Flash incurred USD 7.95 in router inference costs, or approximately USD 0.0022 per scored task. This was about 6% of the task-model spend. We estimate the bill using OpenRouter rates, including discounted rates for cached input tokens.
We cap the router’s token budget at 32,000 tokens per task to make inference costs easier to budget. GPT-5.6 Luna, GPT-5.6 Sol, and DeepSeek-V4.1-Flash achieved similar accuracy on LLMRouterBench, suggesting that smaller models can serve as routing agents in this setting. The tools in Table 1 handle queries in real time at minimal cost, so their runtime and memory overhead are negligible.