- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates silent configuration faults in multi-agent LLM evaluation through a case study of a five-role pipeline. The authors identify seven faults involving prompt handling, decoding parameters, token limits, grading, and dataset selection, showing how such errors can lead to misleading experimental conclusions. They propose several detection checks and a recording proxy, and report corrected results on two models and three benchmarks. The paper also examines verification-based repair and its effectiveness relative to multi-agent pipelines.
Critical strengths
- A relevant problem supported by concrete failure cases. The paper demonstrates how silent implementation errors can substantially alter evaluation results. Several cases are supported by convincing diagnostic evidence, making the observations practically useful.
Critical weaknesses
-
Limited methodological novelty and insufficient validation of the proposed detectors. Most detection mechanisms resemble existing logging and sanity-checking practices, while their validation focuses primarily on known faults from a single harness. More importantly, it remains unclear whether the recording proxy can verify parameters after backend processing or whether the detectors can reliably identify unseen failures.
-
Insufficient experimental support for the broader conclusions. The verifier-independence analysis confounds verification mechanisms with task domains, making causal attribution difficult. The corrected evaluation also retains an acknowledged answer-extraction bias, while the overall findings remain limited to one audited pipeline and two small models.
Questions for authors
-
What is the main contribution beyond existing evaluation logging, configuration validation, and sanity checks? How do the proposed detectors differ from established evaluation tools?
-
Can the authors evaluate the detectors on unseen faults through controlled fault injection and report false-positive and false-negative rates?
-
Does the proxy observe actual model inputs and effective decoding parameters after backend processing, or only incoming API requests? How can it detect prompts discarded by the backend's chat template?
-
Can the authors compare independent and model-generated verification on the same tasks and initial solutions to isolate the effect of verifier independence?
-
The evaluation relies on relatively dated models, Qwen2.5-7B and Llama-3.1-8B. Could the authors extend their experiments to more recent models of comparable size, such as Qwen3-8B or Qwen3.5-9B, to examine whether the conclusions about multi-agent performance and verifier-based repair still hold?
Revised PDF and supplementary material
We have uploaded the revised PDF and a revised supplementary archive. Appendix H lists every change from the submitted version. Every table can be regenerated with the commands in the archive's README and the numbers in the text checked against their output. The archive includes the key file and the frozen code, so both hashes in our first reply can be checked:
e1de82d3...is the SHA-256 ofrebuttal/blind/key.jsonandea1945b6...is the SHA-256 ofrebuttal/blind/frozen_manifest.sha256.Blind fault injection (§5.4, Tables 2 and 6). Scored once against the sealed key under the rule we posted:
The detectors catch a declared parameter that never left the framework, with no false alarms. They are blind to what a backend does after receiving a correct request, as we named in advance. The one framework-side miss was a null injection. Two partial backend system-prompt drops are visible in the raw counts but fall below the frozen rule. We did not change the rule after unsealing.
A fault in our own repair loop, found by the seed check (§5.2, Table 7). At temperature 0 every single-call cell we tested was byte-identical across seeds and across a same-seed rerun. Execution repair on MBPP was not: the test output that becomes the repair prompt quoted a random temporary-file path, so repair prompts differed between runs. We fixed it and reran all 14 execution-repair cells (Table 8). No difference from the single call changed sign and no conclusion changed, but some numbers we posted moved:
What else our first reply promised and where it is. Response-side proxy layer and the F1 reproduction: §5.3 and Table 5. Framework-trace evidence: withdrawn in §5.3, with specificity now measured on the 60 clean runs. Stored-results detectors on the submitted cells (0 FAIL, 35 WARN): Table 11. Stage-wise accuracy, including how often the downstream roles change answers that were already right (PBeh Q5: they rewrite 1,289 of 2,522 and break 299): Table 10. The GSM8K grading estimate in §7 is corrected to the measured bounds (14.7 and 15.0 points for the five-role text pipelines, 1.7 and 2.3 for the single call). Checking the text against the data also caught an error from the submitted version: pipelines cost 4 to 19 times a single call, not 4 to 17.
A correction to our reply to PBeh (Q4). We wrote that confidence is rarely emitted. That holds for the final role in the text pipelines (under 4% of calls) but not for the solver, which states one on 63–99% of calls. Its value does not separate right from wrong answers (0.95 or above on 86.6% of correct and 86.3% of wrong answers), so the conclusion stands: it cannot gate anything (Table 12).