acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Verdict: What Accuracy Misses When Choosing LLM Judges for Agent Evaluation

Abstract

A coding agent can skip a required action and still claim that it completed it. A judge may accept that claim even though the trajectory shows otherwise. More broadly, when one judge looks better than another, a single accuracy score does not tell us why. The difference may come from the decision cutoff, from how the judge handles misleading trajectory content, or from useful information that never reaches its final verdict. We introduce the *Verdict Audit* to separate these possibilities. We apply it to two practical judge-selection settings. For Qwen3-8B and Qwen3-14B, a large accuracy gap mostly disappears after adjusting the PASS/FAIL cutoff, but the smaller model remains more vulnerable to false completion claims when reasoning is enabled. For Qwen3.6-27B and Gemma 4 31B, Gemma 4 makes better output-level rankings, while Qwen3.6 contains useful error information internally that its final verdict does not use. Reading that information narrows the gap, but the fixes we test do not reliably correct the errors. The result is simple: choosing a judge requires understanding why its decisions differ, not just which model has the higher score.

Primary area:
interpretability and explainable AI
Keywords:
LLM-as-a-Judge, Agentic Evaluation, Evaluation Reliability, Judge Robustness, Decision Calibration, Internal Representations, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_K9rMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Verdict Audit is a four-stage procedure for comparing LLM judges on agent trajectories. It examines output ranking and threshold choice, robustness to misleading trajectory content, information recoverable from hidden states, and whether interventions can selectively correct errors. Threshold tuning substantially reduces the balanced-accuracy gap between Qwen3-8B and Qwen3-14B, while stress tests reveal robustness differences whose ordering reverses across domains. For Qwen3.6-27B, hidden-state probes improve ranking over the verdict margin, but neither a balanced-accuracy gain nor selective correction is established.

The paper presents useful robustness findings and interprets its statistical results carefully. However, the added model-selection value of the internal stages remains insufficiently demonstrated, the probing comparison is limited by the chosen output readout, and important experimental details are missing. Together, these issues make the contribution of the full four-stage procedure less convincing than the output-level and robustness analyses on their own.

Critical strengths

The paper addresses a practical problem: selecting judges for long agent trajectories when aggregate accuracy can obscure important differences. The controlled false-completion experiments are useful, particularly the reversal of relative vulnerability between RuVerBench and AgentRewardBench.

The statistical interpretation is generally careful. The authors distinguish a failure to detect a difference from evidence of equivalence, acknowledge limited power, and separate readable internal information from deployable error correction. Trajectory-grouped evaluation and matched intervention controls are also strengths.

Critical weaknesses

The added decision value and practical scope of the full audit remain unclear.

  1. The paper presents Verdict Audit as a procedure for choosing judges, but the clearest actionable findings come from Stages 1 and 2: threshold tuning changes the performance comparison, and stress tests reveal robustness tradeoffs. Table 3 does not demonstrate a deployment decision that changes because of Stages 3 or 4. While Stage 3 identifies additional information in Qwen3.6, Stage 4 does not establish selective error correction.

More importantly, the audit focuses exclusively on binary PASS/FAIL verdicts. In practical agent evaluation, knowing that a task failed is often insufficient. Understanding what went wrong, where the failure occurred, and why can be much more useful. Related work such as TRAIL, Who&When, and AdaMAST explores error taxonomies and failure attribution. Although these are not direct baselines, the paper does not discuss how Verdict Audit relates to such diagnostic approaches or whether it could evaluate more informative judgments. Given that the individual techniques are already established, the limited demonstrated benefit for judge selection and the narrow focus on binary verdicts weaken the contribution of the proposed framework.

  1. The probe results support a narrower conclusion than a general gap between internal and output-level judgment quality. Section 4.3 constructs the output score from first-token probabilities for fixed verdict labels. The probe is therefore compared with one particular output readout. The conditional analysis and matched label-budget experiments are useful, but do not establish whether the advantage persists with alternative prompts, verdict labels, full-label sequence scoring, or trained output-only baselines using richer output features. The evidence supports information beyond the chosen margin; it leaves open how much of that advantage depends on the output scoring protocol.

  2. The experiments do not resolve the practical value of the internal analyses with sufficient precision. At a label budget of 200, the reported balanced-accuracy change from adding the Qwen3.6 probe is +0.073, with a confidence interval of [−0.050, +0.184]. The paper states that the design can detect gains of roughly 0.15–0.23, leaving smaller but potentially useful improvements unresolved. Thus, the lack of an established gain should not be read as evidence that probing has little practical value; rather, the experiments are not precise enough to settle that question. This limits the strength of the deployment conclusions drawn from the internal stages. AgentRewardBench provides useful cross-domain evidence for selected Case A stress tests, but does not address this limitation for Case B.

  3. The experimental specification has major reproducibility gaps. Section 3.4 states that the exact sampled interchange layers are reported in the appendix, but I could not locate that list in Appendix B.8. Appendix B.4 describes the probe family and grouped evaluation, but does not provide enough detail to reconstruct the candidate sites and regularization-selection procedure. Table 10 reports cohort sizes without identifying the exact items or the number of unique trajectories, which matters because checks from the same trajectory are dependent. I also could not find a repository or experimental-artifact link in the provided submission. These omissions make it difficult to reproduce the comparisons and assess their effective sample sizes.

Questions for authors
  1. Table 3 describes how each stage changes the interpretation of the models. Can you identify a concrete deployment decision that changes after Stages 3 or 4, beyond the evidence from threshold tuning and stress tests? What would justify their additional cost?

  2. Does the Qwen 3.6 probe advantage persist under alternative prompts and verdict labels, full-label sequence scoring, and trained output-only baselines evaluated with the same trajectory splits and label budgets?

  3. Given the reported detectable balanced-accuracy gains of approximately 0.15–0.23, what practically relevant effect size was the study intended to resolve? Could additional evaluation narrow the uncertainty around the probe’s decision-level benefit?

  4. Could you provide the exact interchange layers, candidate probe sites, regularization-selection procedure, prompts, seeds, and intervention configurations? Section 3.4 explicitly points to an appendix layer specification that I could not locate.

  5. Could you provide an anonymized artifact link with evaluation scripts, item IDs, trajectory IDs, and split assignments for the reported cohorts, including the 700 rubric checks, 73 shared omission cases, and 200 AgentRewardBench items?

  6. Could Verdict Audit be extended to evaluate error categories and failure localization, rather than only binary PASS/FAIL decisions? How do the authors see its relationship to approaches such as TRAIL, Who&When, and AdaMAST?

3 comments

Thank you for recognizing our robustness findings and statistical controls. We have substantially revised the Introduction, Related Work, Results, and appendices to address your concerns.

1. Why Stages 3–4 matter. Stages 1–2 help teams choose a judge and test whether its decisions follow recorded evidence. But when mistakes remain, teams face another decision. Can internal information help detect those mistakes, and can it be used to correct them? Stage 3 tests whether internal states identify errors beyond what the outputs already reveal. Stage 4 tests whether interventions correct wrong decisions without disrupting correct ones.

Our experiments show why these distinctions matter. Qwen3.6 contains recoverable information, but its probe does not establish better pooled error detection than accessible outputs. Steering also changes correct and incorrect PASS judgments together. Neither result justifies deploying the tested additions. The later stages therefore help teams avoid investing in internal methods without evidence of additional benefit. They become relevant when a team considers such methods, rather than being mandatory for every evaluation (Sections 4.3–4.4, Discussion).

2. Stronger output baselines. We added predictors using the verdict margin, input length, and rubric category on identical trajectory-grouped splits. For Qwen3.6, the strongest accessible baseline achieves AUC 0.745 versus 0.797 for the probe. The paired difference of +0.050 [−0.004, +0.101] does not establish an overall advantage (Appendix F, Table 8). We do not extend this conclusion to untested prompts, labels, or full-label scoring.

3. Statistical precision. At 200 labels, the estimated balanced-accuracy gain is +0.073 [−0.050, +0.184]. Our added power analysis shows detectable gains of approximately 0.15–0.23. Smaller useful improvements remain unresolved and would require more independent trajectories. We do not interpret this as evidence that probing cannot help (Appendix F).

4–5. Reproducibility. Appendix B now specifies the exact interchange-layer grids, fixed probe depths, regularization penalty C=0.05, and grouped fitting. Appendix F provides intervention sites and controls. Appendix A documents the 700 checks from 189 trajectories, 73 omission checks from 53 trajectories, and 200 AgentRewardBench checks from 188 trajectories. We have also uploaded anonymized code with prompts, configurations, seeds, item and trajectory IDs, and split assignments.

6. Failure localization. TRAIL, Who&When, and AdaMAST study where and why an agent failed. Verdict Audit asks whether the judge recognized and used evidence of that failure. These are complementary questions, as explained in the revised Related Work. Categorical and localized judgments are possible extensions, but are not claimed as validated here.

We hope the additional experiments, clearer methodological details, and revised discussion address your concerns about the value of the internal stages, the strength of our comparisons, and reproducibility. We appreciate your consideration of these revisions.

Thank you for your reply and the work you have done. I have reviewed the attached materials and can say the following:

  1. The revised explanation makes the role of Stages 3 - 4 clearer: they help assess whether internal methods are worth adding to the evaluation pipeline. This partly addresses my concern. However, the reported total of $592.70 does not show how much these stages add to the cost. Could you provide a separate estimate for Stages 3-4?
  2. Stronger output baselines. The baseline with margin, input length, and rubric category has indeed been added, and its AUC of 0.745 can be reproduced from the archive.
  3. The issue of statistical precision remains: the interval [−0.050, +0.184] is too wide to determine the practical benefit of the probe. No additional data has been provided to resolve smaller effects.
  4. The main missing experimental settings are now provided, and I consider this issue resolved.
  5. Reproducibility. The archive largely addresses my concern: 115 reported quantities can be recomputed from the attached data, and identifiers and split assignments are available. However, 42 quantities are provided as saved analysis outputs, probe activations are not included, and the exact checkpoint revision of Qwen 3.6 is unknown. Therefore, full reproduction of experiments remains limited. Please explicitly state these limitations in the paper and soften the wording about full reproducibility.
  6. Connection with failure localization. I consider this issue closed; extending the approach to failure localization remains a possible direction for future work.

Thank you for reviewing the revised materials and confirming that several concerns have been resolved. We have added further analyses and documentation to address your remaining questions.

1. Cost of Stages 3–4. Stage 3 reuses activations collected during Stage 1, so probe fitting requires no additional GPU inference. For Stage 4, our experimental records identify approximately $115.18 spent on causal interventions across the four judges, included within the earlier campaign total. Appendix G now reports this breakdown.

3. Statistical precision. Your concern was whether the available evidence could establish a practical improvement from probing. We revisited the existing label-budget experiments across all four judges and accounted for dependence across their 189 shared trajectories.

The dependence-adjusted pooled balanced-accuracy difference is −0.008 [−0.077, +0.060], improving precision by 1.70 times over the original Qwen3.6 estimate. Three of four judges have negative point estimates. This provides stronger evidence that recoverable internal information does not consistently translate into better decisions across these judges.

A smaller gain of +0.05 remains possible. We therefore quantified the evidence needed to detect such an improvement. Our 80%-power calculation estimates approximately 767 labeled checks per judge, compared with the 200 used. Appendix F now presents the pooled analysis, dependence adjustment, and the assumptions behind this sample-size estimate. The new analysis gives us a more precise result from existing data and establishes the scale of evaluation needed to resolve smaller effects.

5. Reproducibility. We have expanded the supplementary archive to include the original verdict-position activations for all four judges, alongside experimental scripts, prompts, configurations, item and trajectory IDs, and split assignments. Reviewers can now rerun the probe comparisons from saved model states rather than relying solely on reported analysis outputs. We have also refined the manuscript's Reproducibility Statement to distinguish the analyses supported by these materials from the remaining requirements for full GPU reproduction.

We hope these additional results and materials address your remaining concerns about experimental cost, statistical precision, and reproducibility. We appreciate your careful review and consideration of these revisions.

Reviewer_6wqyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a 4-stage evaluation protocol for LLM as judges of agentic trajectories, evaluating different aspects of a model beyond aggregate accuracy. Stage 1 test AUC and threshold tuning; stage 2 claim injection and context padding; stage 3 hidden-states probes; stage 4 steering. The protocol is used to evaluate Qwen3-8B, Qwen3-14B, Qwen3.6-27B, and Gemma 4 31B on RuVerBench AgenticCoding dataset.

Critical strengths

Claim Injection with counterfactual design in the context of agentic coding appears to indeed be novel. It also leads to an unexpected result - Qwen3.6 and Gemma 4 move toward FAIL.

Good study design from the statistical point of view: clustering the data by trajectories, honest reporting of which results are (in)significant, equal-norm random-direction controls. Reporting the result on AgentRewardBench which points in the opposite direction of the main dataset is commendable.

Critical weaknesses

Limited novelty

Stage 1. Using token probabilities as verdict score is not novel (Liu et al. 2023). Threshold tuning is well-known, the paper acknowledges this in “We therefore use held-out decision-threshold tuning diagnostically, rather than proposing a new calibration method, to test whether a candidate-model comparison survives adjustment of the decision boundary.”

Stage 2. Context padding is well-known (Shi et al. 2023).

Stage 3. Linear probes are well-known, for example the appropriately cited (Orgad et al. 2025)

Tiny scope and inconclusive results

The paper positions itself to answer a practical question (L40): when one judge looks worse than another, or two candidates look broadly similar, what does a team need to know before choosing between them?

It does not come up with a satisfactory answer.

The paper presents an empirical evaluation of 4 models on ≤210 agentic trajectories (dataset size per the RuVerBench paper) with 117 FAIL checks among the 700 used. This understandably leads to a plethora of statistically insignificant results. The results which are significant are still mostly diagnostic and with lower confidence bounds close to zero: 0.029 for Gemma vs Qwen AUC, 0.018 for probe AUC gain, 0.02 for claim detection gap. Claim Injection effect is more pronounced - but it’s a synthetic perturbation, not an organic agent failure.

The results are tied to the specific dataset. This is especially relevant for things like threshold tuning and internal readout which are tuned and then evaluated in-domain.

This, together with model choice, severely constraints the practical utility. The submission analyses fine differences between models with ≤31B parameters for a task where they are weak in absolute terms (balanced accuracy ≤0.7). Most practitioners will just go for a larger stronger model, at RuVerBench frontier models score 0.85-0.89 (under a slightly different protocol),

At a higher level, the idea that LLM judges’ behaviour differs beyond the aggregate scores is well-established, for example, the RuVerBench papers makes this observation. Verdict Audit runs the models through a gauntlet of evaluations, but answering a practical decision-making requires tying those proxy metrics to downstream utility, which is absent.

Questions for authors

Please clearly enumerate the novel contributions the paper makes. Please clearly attribute all the methods reused from the prior work.

Generic Claims are introduced in section 3.2, Direction Geometry in section 3.4 - but no results are reported. Were experiments run with them?

Figure 3 in the RuVerBench paper states there are 843 rubrics in AgenticCoding, how were the 700 rubric checks selected in the submission?

Line 383 states “Qwen3.6 also does not meet the preregistered pre-final recovery criterion” - where is the preregistered criterion documented?

3 comments

Thank you for recognizing our counterfactual experiments and statistical controls. We have substantially revised the Introduction, Related Work, and Discussion to clarify our contribution.

Novelty and practical value. A strong benchmark score does not necessarily mean a judge catches the failures that matter. For example, Qwen3-8B and Qwen3-14B achieve nearly identical tuned accuracy (0.641 and 0.639), yet catch 9% and 62% of omissions at their original cutoffs. A team choosing between them based on accuracy alone could miss this difference. Verdict Audit tests four connected questions about decision quality, evidence use, internal information, and selective correction. Each stage establishes what the previous result cannot, helping teams determine what evidence they need before selecting a judge or adopting an improvement. This is the contribution beyond the individual methods. These findings directly inform judge selection and whether additional diagnostics or corrections are justified. Measuring downstream gains from adopting those decisions is a separate question.

Why these models and tasks. We deliberately compare similar-sized and same-family judges because these are realistic choices evaluation teams face. The Qwen comparison reveals differences hidden by similar scores, while Qwen3-14B and Gemma 4 show opposing responses to false completion claims. Simply choosing a larger model does not establish whether its verdicts follow recorded evidence. We use verifiable binary requirements to keep ground truth fixed during perturbations and open-weight models to test all four stages on the same systems.

Additional evidence. Our new paired experiment shows that Qwen3.6 and Gemma 4 no longer exhibit the earlier toward-FAIL effects under natural agent wording. The omitted actions remain unchanged, showing that the wording of an evaluation can change its robustness conclusion (Figure 2, Appendix D).

Your specific questions. Generic, negated, and unrelated claim controls are reported in Appendix C, and direction geometry in Appendix F. Appendix A traces 843 rubrics to 700 after 88 length and 55 size-cap exclusions, retaining 189 trajectories. We also corrected the recovery criterion from "preregistered" to post-hoc and report no efficacy estimate for the unfinished follow-up (Appendix G).

We hope the revised manuscript and additional experiments address your concerns about novelty, experimental choices, and practical value. We appreciate your consideration of these changes.

Thank you for providing the explanations and the clarifications.

My original weaknesses stand - combining established techniques with small improvements constitutes minimal novelty; the empirical study's scope is tiny, the results are not nearly enough to provide practical guidance to LLM users.

On top of this, the original paper claimed a non-existent preregisteration. I appreciate the correction, but this was misleading.

Thank you for revisiting the manuscript. We would like to clarify our contribution and why our experimental scope is appropriate for the questions we ask.

Scientific contribution. We ask what evidence an evaluation team needs before selecting a judge, trusting its decisions, or adopting an internal improvement. Verdict Audit addresses this through four connected stages, each testing something the previous stage cannot establish (Introduction, Table 1, Discussion).

We are not proposing a new benchmark or trying to outperform existing evaluation methods. We use these methods to investigate whether their results actually justify the conclusions evaluators draw from them.

For example, two judges with nearly identical tuned accuracy detect 9% and 62% of known omissions at their original cutoffs. Claim sensitivity reverses across domains, and natural wording changes robustness conclusions even when the omitted action stays fixed. Recovering internal information also does not necessarily improve error detection or correction. These findings explain why evaluators need to look beyond accuracy and check what their results actually establish.

Experimental scope. We deliberately chose comparable Qwen judges within the same family and Gemma for cross-family comparisons. Section 3 explains the relevance of open-weight evaluators, citing Kim et al. (2024) and Huang et al. (2025). Verifiable binary requirements and matched controls isolate the effects we study, while open weights allow us to investigate all four stages on the same systems. A larger benchmark would test how widely our findings generalize. Our experiments instead test whether the underlying assumptions hold under controlled conditions.

Preregistration. Thank you for identifying the incorrect terminology. We corrected "preregistered" to "post-hoc" and documented the chronology in Appendix G. Our central findings do not depend on this criterion, which concerned an unfinished follow-up for which we claim no efficacy. The anonymized reproducibility archive provides code, configurations, and supporting evidence for independent verification.

We hope this clarifies our scientific contribution and experimental choices. We appreciate your consideration of the revised evidence.

Reviewer_zgimOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows that choosing an LLM as a judge system for agent trajectory evaluation needs a more comprehensive system instead of only the eaccuracy. It proposes a four-stage Verdict Audit system. Stage 1 measures threshold independent ranking with verdict AUC. Stage 2 applies stress tests, namely false completion claim injection with a negated claim, a generic claim and an unrelated control, and context padding. Stage 3 fits linear probes on the verdict position residual state and checks whether the probe adds information beyond the verdict margin. Stage 4 tests whether steering, claim masking and interchange can selectively correct false PASS errors.

Critical strengths

The control discipline. The false claim is compared with an unrelated claim of similar length, steering is compared with an equal norm random direction, claim masking is compared with length matched removal of non claim text, and the Case A equivalence margin is fixed in advance. This is what makes the negative results credible, and negative results in this area are usually not credible.

The empirical claim that one accuracy gap mixes threshold placement with robustness, and that the robustness ordering of the same two judges reverses between RuVerBench and AgentRewardBench. This is the finding I would cite, because it directly attacks the common practice of picking a judge by a leaderboard score and it is backed by a paired difference with an interval.

Critical weaknesses
  1. The relation to prior work is not fully discussed. All four stages are standard practice, AUC and threshold tuning come from classifier evaluation, claim injection is close to existing prompt injection and sycophancy work, and the probe and steering stages follow Representation-as-a-Judge and representation engineering. The paper cites these contents, but does not make a fair comparison. It is hard to see what the audit adds beyond assembling known diagnostics. Concretely, Case A concludes that the gap is a threshold artifact, but the paper does not compare threshold tuning with temperature scaling or isotonic regression, and it does not show whether a calibrated probability leads to the same conclusion. Secondly, the study does not compare its findings with the meta evaluation reported in the RuVerBench or AgentRewardBench papers themselves, so it is unclear whether the false claim vulnerability or the internal signal is new.

  2. The scale is small and the two cases are chosen by the authors, so the audit is illustrated rather than validated. Four open weight judges are used and no frontier closed model, although teams deploying judges mostly use closed APIs. The main table rests on 700 rubric checks, and the key robustness and masking analyses rest on 73, 84 and 39 items. The primary domain is one benchmark family, with a second domain used only for selected Case A comparisons. Every judgment is a binary rubric check on a single trajectory, with no pairwise comparison, no graded scale and no multi requirement aggregation. There is no repeated sampling over different data splits, so the stability of the reported intervals across splits is unknown, and there is no check of human agreement on the gold labels. There is also no cost accounting, which matters because the internal stages require activation access, labeled data and extra inference. Since currently LLM judging costs have become a large number in especially the agent tasks, this is an major issue.

The audit ends without a usable remedy. Every tested fix fails, the internal signal is readable but not actionable, and the pre registered subspace follow up was not run. After four stages the practical guidance is to look at more than one number, which is reasonable but thin. Combined with the power limits that the authors report, roughly 0.15 to 0.23 on balanced accuracy, the paper cannot distinguish a genuine absence of a usable signal from a signal that the design is too small to detect.

Questions for authors
  1. Would temperature scaling or isotonic regression on the same training split lead to the same conclusion as threshold tuning in Case A. If the conclusion depends on the recalibration method, the framing of the gap as a threshold artifact becomes weaker. Please discuss on it.

  2. With 73 omission items, what is the minimum detectable Claim Effect at 80% power? I would like to see this next to Table 4 so the reader can judge the reversal across domains.

3/ The False Claim is appended to the closing message. What is the base rate of completion claims in RuVerBench to begin with? If natural false claims are much subtler than the injected sentence, does the effect survive a more realistic phrasing?

Update in 2026/11/16: Since the authors do not give any feedback so far, I tend to not change my scores.

1 comment

Thank you for recognizing our controls and cross-domain reversal. We have substantially revised the Introduction, Related Work, and Discussion to clarify our contribution.

Novelty. A strong benchmark score does not necessarily mean a judge catches the failures that matter. For example, Qwen3-14B and Qwen3-8B achieve nearly identical tuned balanced accuracy (0.639 and 0.641), yet detect 62% and 9% of omissions at their original cutoffs. Similarly, finding information inside a model does not establish that it can improve decisions. Unlike benchmark evaluations that primarily measure judge performance, Verdict Audit tests whether the available evidence supports the decisions an evaluation team needs to make. Its four connected stages examine conclusions that the previous results cannot justify. Our experiments show where these assumptions fail and what additional evidence is needed before selecting a judge or adopting an improvement. This is why the stages matter together, beyond the individual methods.

Experimental choices. We chose binary action requirements because the recorded actions provide verifiable ground truth, allowing us to change completion claims without changing the outcome. We chose four open-weight judges across different sizes and families because they are practical evaluation options and allow us to study the complete chain, including internal states and interventions, which closed APIs cannot support. AgentRewardBench provides a second domain to test transfer.

Your questions

  1. Calibration. Temperature scaling and isotonic regression preserve the tuned comparison (Appendix C).
  2. Power. We added 80%-power detectable effects of 0.876–1.235 FAIL-margin logits without reasoning (Appendix F).
  3. Realistic claims. This changes an important earlier finding. Under natural wording, Qwen3.6 and Gemma 4 no longer show the earlier toward-FAIL effects, while Qwen3-14B shifts another −1.18 logits [−2.04, −0.32] toward incorrect PASS. The omitted action remains fixed, showing that test wording can change the robustness conclusion (Figure 2, Appendix D). We also screened naturally occurring claims and found 28 candidates, but none could be verified as targeting the omitted requirement, so we do not estimate a base rate.

We additionally document label review, uncertainty, compute costs, and provide anonymized code.

We hope the revised manuscript and additional experiments address your concerns about novelty, experimental design, and robustness. We appreciate your consideration of these changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.