- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces dynamic task routing, an evaluation setting in which a system must select the task document valid at a requested time before answering it. The authors construct 31,119 episodes from six existing benchmarks, with 373,428 queries covering task changes, preservation, propagation, and historical queries. Task contents remain fixed; events change their temporal bindings. The proposed RIAG system combines a deterministic resolver, version-keyed caching, and two initially unexposed reasoning agents, followed by conditional audit and repair.
The evaluation compares RIAG with several multi-agent methods and reports 54.24% joint routing-and-answer accuracy at 0.62 calls/query for homogeneous RIAG, compared with 32.22% at 18 calls/query for Debate. Additional MATH experiments examine resolver transfer, inference settings, cache reuse, and response diversity.
Critical strengths
The evaluation targets a meaningful gap in task-based reasoning benchmarks. A system operating over an event history must distinguish updates that change its task from events that should leave prior work unaffected. The protocol makes this distinction explicit and also tests how retractions affect historical queries. Separately scoring document selection is important: the paper shows that different documents can share the same gold answer string, so answer-only evaluation can hide an incorrect task binding. I regard this evaluation design as the paper’s main contribution.
The experiments show that this distinction matters for the evaluated methods. In Table 1, the comparison methods select the correct document on only about 59–64% of queries, despite using multiple model calls. Their performance also falls substantially between baseline and historical queries. These results identify a concrete weakness of the tested multi-agent pipelines, rather than simply proposing another evaluation category. The fact that a deterministic resolver handles the routing successfully does not diminish this finding: it provides a useful reference and shows that more deliberation is not a substitute for appropriate state handling in these configurations.
Critical weaknesses
The benchmark would benefit from a simpler, matched reference baseline. RIAG’s reasoning agents receive an already resolved task, whereas conventional baselines receive the event history and must select the document themselves. RIAG also reuses cached answers. This is a legitimate whole-system comparison, but it leaves open how much of RIAG is needed to obtain the reported benefits. A resolver + cache + single LLM baseline is particularly important here, since each episode contains only three immutable documents and twelve queries. Its accuracy is unknown, but it would retain the basic routing and reuse mechanisms. The resolver-controlled GoA experiment helps, although it also changes the agent configuration and decoding. These omissions limit the conclusions about RIAG’s reasoning procedure; they do not, in my view, invalidate the evaluation contribution.
The recovery procedure makes the main results difficult to compare consistently. Appendix D.3 states that the homogeneous result includes 34,796 recovery calls replacing 20,739 parse-failed queries, while the heterogeneous result retains 4,453 failures from its pre-fix run. The extra calls are included in the reported cost, but the two configurations were still evaluated under different recovery procedures. It is also unclear how the combined original and recovery executions relate to the stated four-call-per-key limit, given that failed calls are supposed to consume budget and failed results are cached. This needs clarification before the homogeneous–heterogeneous difference and the production-run budget can be interpreted confidently.
Questions for authors
-
Matched reference baseline. Are results available for a single LLM using the same resolver, cache, and output handling as RIAG? This comparison would help establish a useful reference for the benchmark and show whether the additional reasoning stages improve the accuracy–cost trade-off.
-
Recovery protocol. What exactly was changed and re-executed in the homogeneous recovery run? Why was the heterogeneous configuration not evaluated with the same corrected implementation? Please clarify whether the four-call limit holds when original and recovery calls for the same key are counted together, or whether the guarantee applies only to a separate corrected execution.
-
Metric definitions. Table 5 says that RIAG’s answer and joint accuracy rows are identical because its artifacts record one correctness predicate. Were the two metrics computed separately according to Section 5.1? Also, does a missing or malformed answer count as a selection failure even when the resolver selected the correct document? This affects the interpretation of the routing results.
3 comments
We sincerely appreciate your time and thoughtful evaluation.
Your feedback is invaluable in improving our work. We would be grateful if you could share any further thoughts on our revisions, particularly whether there are any remaining concerns we could address during the discussion period.
Thank you again for your consideration.
Dear Reviewer qcaC,
We wonder if our new results resolve your concerns. If so, we hope you could consider adjusting score if you are comfortable.
Thank you very much.
The author team
Response to Reviewer qcaC
Q1. Matched reference baseline with the same resolver, cache, and output handling
Matched control. We added a matched single-solver reference (Section 5.2, Appendix E.1). On 373,428 queries, Qwen2.5-7B-Instruct uses matched prompts, parsing, decoding, and scoring with a 2,048-token cap. All calls and tokens count toward cost.
Accuracy–cost decomposition. Caching saves 77.1% of calls and tokens (B2 vs. B1), with −0.02 accuracy points (95% CI [−0.36, +0.30]). Full RIAG adds 1.80 points over B2 (CI [+1.19, +2.38]) at 2.26× the calls and 2.45× the tokens. Dataset gains vary (+1.45 MMLU, +2.55 MMLU-Pro, −1.35 GPQA). The full historical-baseline gap cannot be attributed to multi-agent reasoning.
Routing control. On 600 diagnostic episodes with three seeds (21,600 queries per configuration), a history-based document selector followed by the same solver reaches 33.08% joint accuracy versus 52.96% for resolver + single solver without caching, a +19.88-point difference (CI [+17.05, +22.90]). The solver is matched.
Larger task pools. Qwen3.6-27B evaluates 3/6/12-task pools on matched cores with two seeds (Appendix E.4, Table 13). RIAG's core joint accuracy is 93.1/93.8/94.4%, versus 85.4/86.1/84.7% for the cached single solver. The RIAG core K=12 minus K=3 change is +1.39 points (CI [−2.08, +5.56]). Its full-schedule accuracy declines from 93.1% to 89.1%, while the cached solver declines from 85.4% to 83.2%. At K=12, RIAG uses 0.551 calls/query versus 0.250 for the cached solver. Fixed reuse exposure limits generalization.
Output-budget check. In a separate two-seed 27B cohort, increasing the cap to 4,096 reduces RIAG parse failures from 40 to 3 per 864 queries (Appendix E.3, Table 12). RIAG joint accuracy changes from 83.2% to 86.1%, versus 82.6% to 81.4% for the cached solver. RIAG's advantage is +0.58 and +4.75 points, but source-item CIs [−11.40, +12.77] and [−3.10, +21.47] cross zero. Output validity improves, but a robust accuracy gain is unestablished.
Q2. Recovery protocol and the four-call-per-key limit
Updated matched results. We reran both configurations independently on all 373,428 queries with matched inference, a 2,048-token cap, empty caches, no recovery overlay, and all calls charged. Results replace Table 1 and Tables 4–6, with costs in Table 3.
Comparison. Heterogeneous minus homogeneous joint accuracy is −8.33 points (95% CI [−9.27, −7.44]). The three-seed diagnostic difference is −6.15 (CI [−10.70, −1.74]). Homogeneous RIAG uses Qwen2.5-7B-Instruct for all roles. Heterogeneous RIAG uses Qwen2.5-7B-Instruct, Mathstral-7B, Mistral-7B-Instruct, and Qwen2.5-Coder-7B-Instruct for solver, checker, auditor, and repairer. Neither matched full run has parse failures or graphs exceeding four calls.
Historical recovery audit. Earlier homogeneous recovery changed parsing, canonicalization, output instructions, and multiple-choice constraints. It replaced 20,739 failed query outputs using 34,796 retained recovery calls. The heterogeneous row retained 4,453 pre-fix failures despite 8,557 attempted recovery calls. The reason for non-substitution is unrecorded, so this historical asymmetry cannot establish a model-pool effect (Appendix E.2).
Four-call scope. Each individual reasoning graph uses at most four calls, including failures. Combining original and retained recovery executions instead put 7,273 of 85,637 semantic keys above four cumulative calls, up to 80 per key. Historical spending totaled 232,616 homogeneous calls plus 39,087 other unretained recovery calls. The guarantee is per fresh execution, not across independent reruns. Cached failures cannot restart the budget.
Q3. Metric definitions, answer and selection failures
Independent metrics. We rescored saved predictions (Section 5.1, Appendix D.1, Table 5). Selection requires a schema-valid final response with the correct document. Answer accuracy checks the returned answer against the gold task regardless of document label. Joint accuracy requires both. Missing or malformed responses score zero on all three, even if the internal resolver route is correct.
Why answer and joint coincide. The original homogeneous run had 94.45% returned selection versus 100% internal routing because of malformed responses. Both matched RIAG runs have 100% returned selection and no parse failures. Answer and joint coincide because accepted responses carry the correct document, not from a shared scoring predicate.