acceptodds
Under review as a conference paper at ICLR 2027

When the Harness Confirms the Hypothesis: Silent Configuration Faults in Multi-Agent LLM Evaluation

Abstract

Reported gains and losses from multi-agent LLM pipelines are only as trustworthy as the harness that produced them. We audit one multi-agent evaluation end to end and find seven faults that produced no error and no crash, but changed conclusions: a chat template that silently discarded every system prompt, leaving two condi- tions differing only in that prompt byte-identical on 721 of 721 problems; a gram- mar argument swallowed by a constructor, making the “grammar-constrained” condition a byte-identical duplicate of the unconstrained one; a decoding tempera- ture that never reached the sampler, so a run recorded as greedy sampled at 0.2–0.6 against genuinely greedy baselines; hidden per-role token caps truncating one role on 77–90% of problems; and a grader deviating from the benchmark’s standard protocol, worth up to 11.6 points and biased toward one condition. Every fault produced plausible results that confirmed the hypothesis under test, which is why each survived days of monitoring. We characterise the faults, quantify each one’s effect by re-running the affected cells, and release detectors that find them from outside any framework via a recording proxy. We then report corrected results for the pipeline under study on two 7–8B models and three benchmarks.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, evaluation validity, multi-agent systems, reproducibility, structured output, constrained decoding, self-refinement, execution feedback, small language models, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XA9fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates silent configuration faults in multi-agent LLM evaluation through a case study of a five-role pipeline. The authors identify seven faults involving prompt handling, decoding parameters, token limits, grading, and dataset selection, showing how such errors can lead to misleading experimental conclusions. They propose several detection checks and a recording proxy, and report corrected results on two models and three benchmarks. The paper also examines verification-based repair and its effectiveness relative to multi-agent pipelines.

Critical strengths
  • A relevant problem supported by concrete failure cases. The paper demonstrates how silent implementation errors can substantially alter evaluation results. Several cases are supported by convincing diagnostic evidence, making the observations practically useful.
Critical weaknesses
  • Limited methodological novelty and insufficient validation of the proposed detectors. Most detection mechanisms resemble existing logging and sanity-checking practices, while their validation focuses primarily on known faults from a single harness. More importantly, it remains unclear whether the recording proxy can verify parameters after backend processing or whether the detectors can reliably identify unseen failures.

  • Insufficient experimental support for the broader conclusions. The verifier-independence analysis confounds verification mechanisms with task domains, making causal attribution difficult. The corrected evaluation also retains an acknowledged answer-extraction bias, while the overall findings remain limited to one audited pipeline and two small models.

Questions for authors
  1. What is the main contribution beyond existing evaluation logging, configuration validation, and sanity checks? How do the proposed detectors differ from established evaluation tools?

  2. Can the authors evaluate the detectors on unseen faults through controlled fault injection and report false-positive and false-negative rates?

  3. Does the proxy observe actual model inputs and effective decoding parameters after backend processing, or only incoming API requests? How can it detect prompts discarded by the backend's chat template?

  4. Can the authors compare independent and model-generated verification on the same tasks and initial solutions to isolate the effect of verifier independence?

  5. The evaluation relies on relatively dated models, Qwen2.5-7B and Llama-3.1-8B. Could the authors extend their experiments to more recent models of comparable size, such as Qwen3-8B or Qwen3.5-9B, to examine whether the conclusions about multi-agent performance and verifier-based repair still hold?

Reviewer_XXQ5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits a multi-agent LLM evaluation system and identifies seven silent faults which led to misleading experimental results. The paper explains how these errors affected the experiments, introduces several checks to detect similar problems, and reports corrected results using two small LLMs on three benchmarks.

Critical strengths
  1. The paper discusses a practical problem in LLM evaluation. The seven faults show how small implementation errors can lead to wrong experimental results.

  2. The paper provides detailed descriptions of the errors and their fixes. The original and corrected results also helps to understand and analyze the root causes.

Critical weaknesses
  1. Very limited technical contribution. The paper reports debugging issues from a single evaluation system. Several of the seven faults share similar causes: F2–F4 involve missing configuration settings, while F5 and F6 are due to the same missing-context issue. The checks are mostly standard debugging practices. Overall, the technical novelty and science insights are very limited.

  2. The paper has not sufficiently validated its approach. All seven faults are from one evaluation system, and the detectors are tested on these known faults and two frameworks without such errors. This does not show how well the approach can detect new problems in other systems. It is difficult to judge the usefulness of the proposed methods in a broader way.

Questions for authors
  1. What is the main technical contribution beyond standard debugging and configuration checks? How does the suggested approach improve on existing evaluation tools?

  2. Can the authors test the detectors on new errors introduced into other evaluation systems? This would help to understand if the approach is useful beyond these known cases.

  3. In Section 5.3, how can the recording proxy detect errors like F1 if the system prompt is dropped inside the backend after the API request is received? Does the proxy verify the final input actually seen by the model?

Reviewer_PBehOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits a five-stage multi-agent LLM pipeline and identifies several silent configuration faults involving prompts, sampling parameters, token limits, grading, and evaluation sets. The authors quantify how these faults affected the reported outcomes, and provide automated checks. The stated contribution concerns evaluation reliability. However, the extent to which these findings establish a generalizable auditing methodology remains unclear.

Critical strengths
  1. The audit is very thorough and detailed. Several faults materially affect experimental conclusions, particularly the incorrect grading procedure, which changes reported pipeline accuracy by up to 11.6 percentage points, which is substantial.

  2. The proposed checks are practically useful. Verifying the parameters actually received by the model, rather than relying entirely on declared configurations, is very important for evaluation in terms of reliability.

Critical weaknesses
  1. I am not fully convinced about the methodological novelty. Most identified issues are familiar software-testing and LLM-evaluation problems: ignored parameters, incorrect formatting, hidden defaults, and grading errors for example. The detectors primarily use output comparisons, parameter logging, and consistency checks. It is unclear what new general auditing methodology emerges out of this work beyond applying these checks to an implementation. The authors' behavioral findings about prompting, structured outputs, and self-verification also overlap with existing literature.

  2. The validation is limited, and the corrected results lack deeper analysis. All the faults were discovered in a single harness. Testing the detectors against these known faults, while observing no alarms on two other frameworks, does not establish their effectiveness on previously unknown errors. Additionally, although the five-stage pipeline performs considerably worse than a single call, the paper does not systematically establish where accuracy deteriorates. The available confidence signals are also not analyzed systematically.

Questions for authors
  1. Can the authors clarify what the new methodological contribution is relative to existing software-validation and LLM-evaluation practices? In particular, how does the proposed auditing framework differ from standard logging and consistency checks?
  2. Can the detectors identify previously unknown configuration faults in independent evaluation harnesses? Given the two initial false alarms reported in the appendix, can the authors provide a more systematic estimate of false-positive rates?
  3. The appendix discusses intermediate grading of the Implementor's output. Can the authors provide a more detailed stage-wise analysis showing where correct answers become incorrect, or incorrect answers are corrected, particularly during checking and finalization?
  4. Are the confidence scores produced by the different agents predictive of correctness? Would confidence-based gating help reduce unnecessary downstream revisions?
  5. Does the initial task decomposition or reformulation introduce errors or lose relevant information? How often do the Checker and Judge modify already correct solutions unnecessarily?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF and supplementary material

We have uploaded the revised PDF and a revised supplementary archive. Appendix H lists every change from the submitted version. Every table can be regenerated with the commands in the archive's README and the numbers in the text checked against their output. The archive includes the key file and the frozen code, so both hashes in our first reply can be checked: e1de82d3... is the SHA-256 of rebuttal/blind/key.json and ea1945b6... is the SHA-256 of rebuttal/blind/frozen_manifest.sha256.

Blind fault injection (§5.4, Tables 2 and 6). Scored once against the sealed key under the rule we posted:

fault classplacementtrialsdetected
in the paper's classesframework-side98
in the paper's classesbackend-side100
outside the paper's classeseither210
clean (decoys + second clean batch)–600 false FAIL

The detectors catch a declared parameter that never left the framework, with no false alarms. They are blind to what a backend does after receiving a correct request, as we named in advance. The one framework-side miss was a null injection. Two partial backend system-prompt drops are visible in the raw counts but fall below the frozen rule. We did not change the rule after unsealing.

A fault in our own repair loop, found by the seed check (§5.2, Table 7). At temperature 0 every single-call cell we tested was byte-identical across seeds and across a same-seed rerun. Execution repair on MBPP was not: the test output that becomes the repair prompt quoted a random temporary-file path, so repair prompts differed between runs. We fixed it and reran all 14 execution-repair cells (Table 8). No difference from the single call changed sign and no conclusion changed, but some numbers we posted moved:

  • Table R2 (now Table 9): independent check on Qwen MBPP 79.8 → 80.9 (11/0 → 14/0 fixed/broken), Llama HumanEval 70.7 → 71.3 (2/0 → 3/0), Llama MBPP 77.0 → 75.5 (17/0 → 13/0). Self-written check on Qwen HumanEval 6/5 → 7/6, Qwen MBPP 77.8 → 77.0 (8/2 → 7/3), Llama HumanEval 66.5 → 65.9 (7/12 → 5/11), Llama MBPP 68.1 → 66.5 (7/13 → 7/17). Independent false alarms 0–2.3% → 0–1.5%. Tokens 1.14–1.74 / 3.31–3.63 → 1.13–1.72 / 3.28–3.59. McNemar, independent vs self-written: p = 1.00, 0.18, 0.19, < 0.001 → 1.00, 0.006, 0.064, < 0.001. "Breaks 2–13 correct answers per cell" becomes 3–17.
  • Stage-wise totals (Table 10): 298 broken / 230 fixed / net −68 → 299 / 228 / −71.

What else our first reply promised and where it is. Response-side proxy layer and the F1 reproduction: §5.3 and Table 5. Framework-trace evidence: withdrawn in §5.3, with specificity now measured on the 60 clean runs. Stored-results detectors on the submitted cells (0 FAIL, 35 WARN): Table 11. Stage-wise accuracy, including how often the downstream roles change answers that were already right (PBeh Q5: they rewrite 1,289 of 2,522 and break 299): Table 10. The GSM8K grading estimate in §7 is corrected to the measured bounds (14.7 and 15.0 points for the five-role text pipelines, 1.7 and 2.3 for the single call). Checking the text against the data also caught an error from the submitted version: pipelines cost 4 to 19 times a single call, not 4 to 17.

A correction to our reply to PBeh (Q4). We wrote that confidence is rarely emitted. That holds for the final role in the text pipelines (under 4% of calls) but not for the solver, which states one on 63–99% of calls. Its value does not separate right from wrong answers (0.95 or above on 86.6% of correct and 86.3% of wrong answers), so the conclusion stands: it cannot gate anything (Table 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.