acceptodds
Under review as a conference paper at ICLR 2027

Testing Agent Process Verifiers with Counterfactual Instructions

Abstract

Process verifiers guide tool-using agents by ranking candidate actions. However, evaluating a verifier on a single instruction cannot tell whether its preference actually depends on the task requirement. We introduce crossed instruction-trajectory matching: the same two candidate trajectories are evaluated under two instructions that differ in one requirement. A separate action-level predicate recomputes the required argument from the instruction and shared history, and verifies which candidate action is correct. A verifier must reverse its preference to solve the group, ruling out a fixed candidate preference. Using this design, we construct 195 groups from three tool-use benchmarks and evaluate five prompted judges and five trained reward models. On selection-by-attribute groups, 12 of 42 judge-group pairs answered correctly under the original instruction fail under the edited instruction, including seven fixed candidate preferences. Judges with similar crossed accuracy fail in different ways, including fixed preferences, position-following, and ties. Organizing the listed options by item improves all five judges on both diagnosed and held-out groups, whereas simply restating the criterion does not consistently help. Raw-value tables, which group observed values by item without computing attributes, recover 5-10 of 11 held-out groups, and computed tables recover 8-11. Because success requires correct choices under both instructions, this recovery cannot be explained by a stronger fixed preference for one candidate. Finally, we show that evaluation protocol matters: scoring candidates separately can lose distinctions recovered by direct comparison, and aggregating step-level scores by their minimum can erase correct local error detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tool Agent, Process Verifier

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yvfyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a concept called crossed instruction-trajectory groups. Basically, this looks at two possible actions that start off the same but split at one decision. The paper tests these actions under two instructions that differ in a single requirement, aiming for the “right” action to switch depending on the instruction. The idea is that a verifier only gets credit if it picks the correct action in both cases.

Critical strengths

Big strengths:

  1. The crossed design is a smart, clear way to check for fixed candidate biases. The three-level validation and the detailed error lists (Appendix G) make the experiment credible.
  2. Breaking down errors into types (fixed, position, tie, unresolved) reveals that similar scores can hide different problems. Controls—like withholding listings, matching output budgets, paraphrasing, and rerunning for noise—are thoughtfully done.
Critical weaknesses

But there are weak spots:

  1. The supporting data is limited and pretty clustered. Out of 195 groups, most are string matcher-friendly, and the main findings come from just 15 groups across five scenarios (eight of those sharing a key issue). Another 11 groups come from six listings, and there are no error bars on the main results. Some takeaways feel more like input formatting wins; the model can spot the cheapest choice because the table makes it obvious, not necessarily because of deep understanding. There's evidence that judges have trouble reading and comparing long JSON, which is different from the claim that "verifier preferences don’t follow the requirement.” Held-out experiments meant to mimic real-world use (Appendix R) show a mixed bag: some predictions don’t match up, the effects of different rendering choices blend together, and an important best-of-N comparison wasn’t run. Plus, there's no proof that better crossed accuracy leads to better real-world selection.
  2. Some results depend heavily on how scores are tallied. Ties automatically count as failures, so when using a three-level scale (-1, 0, +1) you get a lot of ties (like 51 for Qwen3.5-35B-A3B). This impacts key findings. Switching to mean aggregation brings back most of the lost signal, so a lot depends on the method, not the model. The way candidates are compared (one-by-one or side-by-side) changes what information is available; something the paper points out. Chance rates also move around (1/4 or 1/16) depending on how strict you are, making it hard to compare figures across the paper. The human audit leaves nine rows with no clear majority, and the agreement score is only 0.716, so some test cases (like list-index) look ambiguous. Overall, the main text feels jammed with detail and leans hard on the appendices.
Questions for authors

-Can you share per-cluster or bootstrap confidence intervals for the main results (like the 42-to-30 stat and the renderer recovery), treating each listing as a unique sample?

  • Do crossed scores actually predict best-of-N or step-level performance in policy rollouts?
  • How do your aggregation results change if you break ties by counting low-scored steps?
  • If you shuffle the raw tables by item (instead of grouping), does the improvement persist, or does it drop off, basically, is this a true understanding effect or just nice formatting?
  • If you drop the seven groups without a human audit majority, do your main conclusions hold? (Appendix F says yes, but please put this in the main text.)
7 comments

Thank you for the detailed review and concrete suggestions. We have expanded the evaluation beyond the groups used in our diagnosis, added controls for how information is displayed and scores are produced, and tested whether feedback from crossed failures improves performance on an external benchmark. We also clarify the scope of our conclusions below.


W1 & Q1. Additional evaluation and uncertainty across scenario clusters

We test whether the failures observed in the original 15 selection groups also occur in 29 additional groups: 11 held-out selection groups and 18 AppWorld groups. We also remove the two largest scenario clusters to check whether a few heavily represented scenarios drive the result.

Each group is evaluated by five judges. The table counts judgments that are correct under the original instruction and those correct under both instructions; their difference is the additional failures exposed. Counts are summed across judges, so 15 groups yield 75 judgments. Confidence intervals are computed by resampling scenario clusters because multiple groups can share the same scenario or tool listing.

GroupsNumber of groups / clustersCorrect under original instructionCorrect under both instructionsAdditional failures exposed [95% CI]
Original groups15 / 5423012 [1, 25]
Additional groups29 / 111008812 [5, 21]
Combined44 / 1614211824 [10, 41]
Excluding the two largest clusters28 / 14927517 [7, 28]

The counts use the first pairwise call for each judge and group. Confidence intervals come from 5,000 bootstrap resamples of the scenario clusters.

Additional failures persist in groups not used in diagnosis and after removing the two largest clusters, including the shared reservation highlighted by the reviewer. The wide interval for the original set reflects its small number of clusters. The added groups strengthen the evidence, although the limited number of clusters still affects precision.

We compare the original tool output with two alternative representations on 11 held-out selection groups. The raw table organizes observed values by item without computing the selection attributes, while the computed table additionally provides derived attributes used for selection. We estimate gains over the original output using paired bootstrap resampling of scenario clusters.

JudgeOriginal listingRaw table: solved; gain [95% CI]Computed table: solved; gain [95% CI]
deepseek-v4.1-flash2/1110/11; +72.7 [40.0, 100]11/11; +81.8 [55.6, 100]
GPT-5.6 Luna1/115/11; +36.4 [16.7, 62.5]8/11; +63.6 [37.5, 90.0]
GPT-5.6 Terra1/118/11; +63.6 [37.5, 83.3]11/11; +90.9 [62.5, 100]
Qwen3.5-35B-A3B1/118/11; +63.6 [40.0, 90.0]11/11; +90.9 [62.5, 100]
Qwen3.8-27B2/119/11; +63.6 [40.0, 90.0]11/11; +81.8 [55.6, 100]

Gains are percentage-point differences from the original output. Both table representations improve performance for all five judges, with confidence intervals excluding zero. The raw-table gains show that recovery does not require supplying precomputed selection attributes. The additional-group analysis appears in Section 5.1, Table 4, and the renderer confidence intervals are reported in Appendix P, Table 14.

W1 & Q2. Connection to downstream selection

To test whether crossed failures provide useful feedback beyond our benchmark, we add a self-improvement experiment. We generate reflections from these failures, include them in the verifier’s prompt without updating its parameters, and evaluate its accuracy in judging agent steps on ToolPRMBench.

A Qwen3.8-27B model generates reflections for a Qwen3-4B verifier trained on AppWorld, BFCL, and ToolSandbox. In the no-feedback condition, reflections are generated from the verifier's own scores and answers on the training cases, without information about which answers are correct. The other conditions additionally identify cases that fail the instruction-only, trajectory-only, or crossed test. This comparison examines whether feedback from each test improves the usefulness of the generated reflections, rather than comparing the presence and absence of reflections.

We evaluate on 210 ToolPRMBench cases from tasks excluded from our training and development sets. This benchmark is relevant to downstream use: Li et al. [1] report that higher ToolPRMBench accuracy is associated with larger gains from best-of-8 search on GTA and BFCL. The table reports accuracy and the improvement from crossed feedback over each alternative, with 95% confidence intervals.

Reflection fromToolPRMBench accuracy (%)Improvement from crossed feedback (percentage points) [95% CI]
No feedback48.5+4.2 [1.0, 7.5]
Instruction-only failures50.2+2.5 [−0.2, 5.5]
Trajectory-only failures49.0+3.7 [0.6, 6.8]
Crossed failures52.7—

We also evaluate crossed accuracy on 57 held-out development groups:

Crossed feedback compared withImprovement in crossed accuracy (percentage points) [95% CI]
No feedback+12.9 [6.4, 19.9]
Instruction-only feedback+8.8 [1.2, 17.0]
Trajectory-only feedback+9.9 [2.9, 17.5]

On ToolPRMBench, crossed feedback improves accuracy over no feedback and trajectory-only feedback, with confidence intervals excluding zero. Its advantage over instruction-only feedback is less certain. On the development groups, crossed accuracy improves over all three alternatives.

These results show that feedback from crossed failures improves verifier accuracy beyond our constructed examples. The relationship between ToolPRMBench accuracy and search performance reported in [1] provides indirect evidence of practical relevance. We include these results in Section 4.4 and Figure 5, with confidence intervals in Appendix N, Tables 12-13.


[1] ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents.

https://aclanthology.org/2026.findings-acl.602/

W2 & Q3. Aggregation and the tie rule

We agree that this finding concerns how scores for individual steps are combined. Minimum aggregation assigns a trajectory its lowest step label (−1, 0, or +1). Two trajectories can therefore tie even when one contains more steps with low scores. Because unresolved ties count as failures, this can obscure differences between candidates.

Following the reviewer’s suggestion, we first compare the minimum labels. If they tie, we prefer the trajectory with fewer −1 steps, then fewer 0 steps if necessary. The table reports accuracy (%) with tie counts in parentheses for the BFCL setting highlighted by the reviewer and the newly added Gaia2 benchmark (also included in Appendix R.3, Table 26).

JudgeBenchmark (groups)MinimumThen fewer −1 stepsThen fewer 0 stepsMean
deepseek-v4.1-flashBFCL value substitution (100)65 (35)76 (21)78 (13)78 (13)
deepseek-v4.1-flashGaia2 (149)32 (94)56 (43)64 (24)53 (33)
Qwen3.8-27BBFCL value substitution (100)90 (8)93 (5)95 (2)94 (3)
Qwen3.8-27BGaia2 (149)22 (114)41 (70)48 (50)44 (53)
Qwen3.5-35B-A3BBFCL value substitution (100)49 (51)62 (27)64 (24)64 (24)
Qwen3.5-35B-A3BGaia2 (149)22 (113)38 (72)46 (52)43 (57)

For the specific case raised by the reviewer, Qwen3.5 on BFCL, this rule reduces ties from 51 to 24 and raises accuracy from 49% to 64%, matching mean aggregation. Across all combinations of judges and settings evaluated, the complete rule matches or exceeds mean accuracy, although some ties remain.

These results show that counting steps with low scores recovers distinctions lost when only the minimum is retained. We therefore narrow the claim to minimum aggregation with coarse labels and unresolved ties, rather than a limitation of evaluating individual steps in general.

Reviewer_ESdZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on the evaluation of process verifiers in agentic settings. In particular, this work addresses a specific case of "contextual" verification, moving beyond existing settings that primarily measure agreement with correctness labels. Instead, the proposed benchmark focuses on counterfactual instructions, where two instructions define different requirements on two candidate trajectories, which share a prefix and differ in one critical action argument. Experiments span both judge models and trained reward models.

Critical strengths
  • The problem setting is relevant, particularly given the increasing importance of process verifiers for guiding agentic systems, both during inference and learning.
  • The proposed framework can reveal failure cases that existing benchmarks may miss. In particular, when verifiers rely on superficial correlations and/or ignore the task context.
  • The framework can distinguish between different failure modes, providing a more informative characterization.
Critical weaknesses
  • Clarity and presentation. I found the manuscript very difficult to follow. The presentation is dense, and it is hard to fully understand what are the key novel contributions, the reusable components versus the ad-hoc methods and analyses, and the main findings across the numerous experimental settings. I believe the clarity of the manuscript needs substantial improvement.
  • Novelty. The core idea (evaluating model preferences under counterfactual instructions) has been explored in previous works. The adaptation to agentic settings and the additional analyses are valuable, but the overall conceptual novelty appears relatively limited. The authors should better clarify the key novel contributions relative to prior work.
  • Downstream validation. The paper includes some experiments on policy-generated decisions, but it remains unclear to what extent the failure modes identified in the benchmark translate into downstream performance, both in inference and learning settings.
Questions for authors

See weaknessess

4 comments

Thank you for your time for the review, and acknowledging the importance of our framework that can reveal failure cases that existing benchmarks may miss. In our reply, we address clarity, novelty, and downstream validation.

Regarding the presentation, we have organized the paper structure, redrawn the figures, and clarified our contribution.


W1. Clarity and presentation

We revised the introduction to clarify why benchmark success can be mistaken for evidence that a verifier follows task requirements, and how the crossed test provides an additional check. We also reorganized the experiments around three questions: whether the test reveals hidden failures, what judgment errors those failures reveal, and whether feedback from them improves verifier performance.

  • Motivation. A verifier can pass a test through a fixed preference, so success alone may be mistaken for evidence that it can judge whether an action satisfies the task requirement. Our evaluation addresses this gap by checking whether the verifier chooses correctly when a changed requirement makes a different candidate correct.

  • Main contribution. We propose an evaluation method that exposes failures missed by separate tests of instruction and trajectory consistency. Across five judges, 20 to 29% of crossed failures occur on groups that pass both separate tests. We also show that feedback from crossed failures improves verifier performance across all three tests and transfers to ToolPRMBench, demonstrating its usefulness beyond detecting failures.


W2. Novelty compared to related work

A blind spot in separate consistency tests. Prior work already studies reward consistency across related instructions and responses [1]. Its two comparisons check whether the original candidate is preferred under the original instruction and whether that candidate receives a higher score under its corresponding instruction. Passing both does not guarantee correct judgments involving the edited candidate. Our pointwise crossed test checks the remaining comparisons and requires both matching instruction–candidate pairs to score above both mismatching pairs. We clarify this distinction in Section 2 and provide a formal comparison and numerical counterexample in Appendix T.

Verified changes in action correctness. For agent decisions, the instructions differ in one requirement, and the candidate actions share the same history and differ in one argument. We execute all four instruction–trajectory combinations and independently check which divergent action satisfies each instruction. This establishes that the changed requirement reverses the correct candidate, rather than assuming that an instruction edit changes correctness, and complements benchmarks that evaluate existing trajectories or individual steps [2–4].

Evidence that the additional comparisons matter. Across five prompted judges, 20–29% of crossed-test failures pass both separate tests. This finding also persists beyond direct instruction–argument matching: among the 185 groups not solved by our argument matcher, 57 of 251 crossed-test failures still pass both individual tests (22.7%; 95% cluster-bootstrap CI: [17.8, 27.5]). Thus, the identified blind spot is not confined to groups solved by this matcher. We report this analysis in Section 4.2.


[1] The Trickle-down Impact of Reward Inconsistency on RLHF.

https://proceedings.iclr.cc/paper_files/paper/2024/hash/8c976a95df6a229551cd28c76627edc9-Abstract-Conference.html

[2] AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories.

https://arxiv.org/abs/2504.08942

[3] AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents.

https://arxiv.org/abs/2603.14465

[4] ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents.

https://aclanthology.org/2026.findings-acl.602/

W3. Downstream validation

To test whether the failures identified by our evaluation can help improve verifiers, we generate reflections from failures detected by three tests. We include the generated reflections in the verifier’s prompt during evaluation, without updating its parameters. Each test uses a different part of the same setup: two candidate actions and two instructions that differ in one requirement, such that the correct action changes with the instruction.

  • Trajectory-only test: Compare both actions under the original instruction. The verifier passes if it scores the correct action higher.
  • Instruction-only test: Score the original action under both instructions. The verifier passes if it gives the action a higher score under the instruction it satisfies.
  • Crossed test: Score both actions under both instructions. The verifier passes only if both matching combinations score higher than both mismatching combinations.

A Qwen3.8-27B model generates reflections for a Qwen3-4B verifier trained on AppWorld, BFCL, and ToolSandbox. In the no-feedback condition, reflections are generated from the verifier's own scores and answers without information about which answers are correct. The other conditions additionally identify cases that fail one of the three tests described above.

We evaluate every condition using all three tests to check whether reflections from crossed failures also improve performance under the two individual tests. The table reports mean results over three seeds on 57 development groups from tasks excluded from verifier training.

Reflection fromF1Trajectory-only testInstruction-only testCrossed test
No feedback71.180.781.366.1
Instruction-only failures74.889.583.670.2
Trajectory-only failures75.085.482.569.0
Crossed failures77.390.690.178.9

The following table reports the improvement from crossed feedback over each alternative in percentage points, with 95% bootstrap confidence intervals over the development groups.

Crossed feedback compared withF1Trajectory-only testInstruction-only testCrossed test
No feedback+6.3 [2.9, 9.9]+9.9 [2.9, 17.5]+8.8 [2.9, 15.2]+12.9 [6.4, 19.9]
Instruction-only feedback+2.6 [0.0, 5.2]+1.2 [−3.5, 5.8]+6.4 [2.3, 11.1]+8.8 [1.2, 17.0]
Trajectory-only feedback+2.3 [0.2, 4.7]+5.3 [0.0, 10.5]+7.6 [1.2, 14.6]+9.9 [2.9, 17.5]

Crossed feedback achieves the highest mean performance on all four metrics. Compared with no feedback, all improvements have confidence intervals excluding zero. Compared with either source of single-axis feedback, the improvements on the instruction-only and crossed tests also exclude zero, while the differences on the trajectory-only test are less certain. The benefit therefore extends beyond the crossed criterion, although not every comparison establishes a clear advantage.

To assess whether the reflections help beyond our constructed examples, we evaluate on ToolPRMBench [4], a benchmark for judging the correctness of steps taken by tool-using agents. We use 210 cases after excluding tasks used in our training or development sets. This evaluation does not use our crossed construction. It also includes the ToolTalk subset, whose tasks were not used during self-improvement.

Reflection fromToolPRMBench accuracy (%)Improvement from crossed feedback (percentage points) [95% CI]
No feedback48.5+4.2 [1.0, 7.5]
Instruction-only failures50.2+2.5 [−0.2, 5.5]
Trajectory-only failures49.0+3.7 [0.6, 6.8]
Crossed failures52.7—

Reflections generated with crossed-test feedback improve verifier accuracy over both no feedback and trajectory-only feedback, with confidence intervals excluding zero. The advantage over instruction-only feedback remains uncertain. These results show that the feedback helps verifiers judge actions beyond our constructed examples, although they do not directly establish improved task completion by agents using those verifiers. These results are included in Section 4.4 and Figure 5, with confidence intervals in Appendix N, Tables 12-13.

Dear Reviewer ESdZ,

Your review was really helpful in improving our draft. Since the deadline is approaching, could you be able to take a look at our responses and check if your concerns are resolved?

Thank you again for your time and effort in writing the review.

Thank you for the additional analyses and clarifications. In particular, the new downstream evaluation is a valuable addition that addressess some of my initial concerns. Regarding novelty, I believe that the conceptual and methodological advances remain limited with respect to existing approaches. Additionally, while the downstream results are encouraging, they do not establish whether the identified failure modes translate into better agent performance or learning outcomes.

Overall, I raised my score to 2, as I believe the additional analyses/experiments strengthened the manuscript, even though broader limitations are still present.

Reviewer_MtY8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether a process verifier changes its preferred agent action when the instruction criteria changes. They study this with crossed groups across BFCL, toolsandbox, τ2\tau^2 -bench. They are created with counterfactual instruction and with two separate trajectory paths. They also do blinded human audit and eventually build 195 execution-verified groups. With evaluation on five prompted judges and five reward models, the paper show that traditional evaluations can overlook failures such as fixed preferences and position bias. Additionally, they find that structuring tool observations improve verifier accuracy and that scoring methods significantly affect trajectory rankings. Overall, the paper proposes a reliable framework for evaluating agent process verifiers.

Critical strengths
  • The validation setup is thorough. The four cell execution check, blinded human audit (91/100 rows) is valuable. The supported sensitivity experiments by excluding seven groups with uncertain audit labels leaves the main contributions unchanged.
  • The defined metric is an important contribution too, it separates chose the right candidate once from preference entails the requirement, by scoring both presentation orders and decomposes failure into fixed preference, position following, ties and unresolved outputs.
Critical weaknesses
  • Lack of diversity? the dataset contains 195 groups - 139 involve relatively simple stated-value matching. The hidden failures revealed by crossed testing are concentrated in just 15 selection groups, 8 of which share the same reservation. I wonder how well the main findings generalize to more diverse agent tasks.

  • The comparison between pairwise and pointwise evaluation changes not only candidate visibility but also prompt wording and output format. Therefore, the observed pairwise improvements cannot be attributed specifically to witnessing both candidates together. A controlled experiment varying only candidate visibility would strengthen this finding.

  • Baseline reporting seems incomplete. Section 4.1 names Qwen2.5-math-prm-7b as a control and appendix J says web-shepherd-8b was "released and used", yet there is no reported results.

Questions for authors
  • outside selection-by-attribute - does second instruction ever expose a failure that single-instruction evaluation misses, for any judge or trained verifier?
  • how many candidate edits were attempted and how many failed each of C1, C2, C3?
  • do authors have thoughts on isolating the effect of candidate visibility by keeping prompt wording & output format fixed between pairwise and pointwise evaluation?
5 comments

Thank you for your constructive feedback. Your comments were invaluable in improving the quality of our submission.

Regarding the presentation, we have organized the paper structure, redrawn the figures, and clarified our contribution.


W1. Diversity of the evaluation

To expand the diversity of the evaluation, we added new benchmarks, AppWorld and Gaia2, bringing the revised dataset to 321 groups across five benchmarks.

BenchmarkGroupsStated valueLookup or mappingSelection by attributeComputation from observations
BFCL4740700
ToolSandbox4025249
τ²-bench1600115
AppWorld692901822
Gaia2149149000
Total32124393336

The expanded evaluation reveals cases where a verifier chooses correctly under the original instruction but fails the additional checks introduced by changing one requirement. The lower pass rates therefore expose failures missed by evaluation under the original instruction. For example, GPT-5.6 Luna passes 94% of AppWorld groups under the original instruction, but only 87% satisfy the crossed criterion; on Gaia2, the corresponding rates are 66% and 46%. The same pattern appears for trained verifiers: Skywork-Reward-V2 passes 87% versus 61% on AppWorld and 98% versus 81% on Gaia2.


W2. Disentangling the effects of candidate visibility and output format

In Figure 5(a) of the original submission, we compared scoring each action separately with showing the judge both actions and asking it to choose one. This changes both what the judge sees and what it is asked to output, so the comparison alone cannot establish whether seeing the alternative candidate explains the improvement. Our new experiment first tests this directly: we show the alternative candidate while still asking the judge to score only the target action, keeping the number of calls and averaging procedure unchanged. We then examine the effects of requesting scores for both candidates and replacing scores with a direct choice.

We evaluate Gaia2 cases where pointwise scoring failed but pairwise comparison succeeded, selected separately for each judge, alongside the same 15 selection groups for all three judges. All conditions use the same success criterion: the correct candidate must receive the higher score or be selected under each instruction; ties count as failures.

We start by scoring each candidate alone twice and averaging its scores. We use fresh evaluations because some originally failed cases succeed upon reevaluation. We then make three successive changes:

  • Show other candidate: Show the alternative while still scoring only the target candidate, keeping the number of calls and averaging procedure unchanged.
  • Score both candidates: Keep both candidates visible and request a score for each.
  • Choose rather than score: Keep both candidates visible and request a direct choice instead of scores.
JudgeGroupsShow other candidateScore both candidatesChoose rather than scoreScore alone → direct choice
deepseek-v4.1-flashGaia2 (49)+100+237 → 49 (+12)
GPT-5.6 LunaGaia2 (78)+12+22+341 → 78 (+37)
GPT-5.6 TerraGaia2 (75)+14−17+1959 → 75 (+16)
deepseek-v4.1-flashSelection (15)+3−3−512 → 7 (−5)
GPT-5.6 LunaSelection (15)+20−912 → 5 (−7)
GPT-5.6 TerraSelection (15)+1−1−310 → 7 (−3)

The three intermediate columns report changes in the number of successful groups from the preceding condition; the final column compares the starting and final conditions.

Showing the alternative helps in all six comparisons while retaining the same scoring procedure. However, switching from scores to direct choice helps on the selected Gaia2 cases and hurts on Selection. Candidate visibility therefore contributes to the observed difference, while the benefit of changing the output format depends on the evaluated cases.
We include this controlled comparison in Appendix S, Table 27.


W3. Excluded baselines

Thank you for pointing out the inconsistency. Qwen2.5-Math-PRM-7B is designed to evaluate mathematical reasoning, while Web-Shepherd-8B is designed to evaluate web navigation. We excluded them from the final comparison because they are not designed for the API tool tasks evaluated in our study. We have corrected the earlier statement that these models were “released and used” and removed the corresponding baseline descriptions.

The trained verifiers reported in the revision are ToolPRM-GRPO v3 and v4, ToolRM-Gen 4B, Agent-RRM 8B, and Skywork-Reward-V2 8B, each evaluated using its published input format.

Q1. Failures revealed by the crossed test beyond selection tasks

We have confirmed that the second instruction exposes additional failures outside selection groups, for both prompted judges and trained verifiers. The table compares accuracy under the original instruction with crossed accuracy on BFCL and Gaia2, neither of which contains selection groups in our evaluation.

VerifierBFCL (47 groups)Gaia2 (149 groups)
GPT-5.6 Luna96% → 89%66% → 46%
Skywork-Reward-V2 8B100% → 98%98% → 81%
Agent-RRM 8B70% → 38%24% → 5%

All results use pointwise scoring. Each entry shows accuracy under the original instruction followed by accuracy under the full crossed criterion. For example, Agent-RRM answers correctly on 70% of BFCL groups under the original instruction, but only 38% satisfy the crossed criterion. Thus, success under one instruction can conceal failures outside selection tasks, including cases involving relatively simple requirements.


Q2. Statistics of filtered instances by each requirement (C1-C3)

The table counts tried replacement values per benchmark: C1 removes edits that change more than one requirement, and C2 removes edits whose changed call errors on replay. C3 held on every group that passed C1-C2 in the original three benchmarks; AppWorld does not separate C1 from C2, and its 37 C3 failures are groups without an independent action-level check. The gap to the groups in the paper comes from free-text arguments kept as a sensitivity set, role swaps, duplicates, adjudication, and two consistency checks for the new benchmarks.

BenchmarkTriedFailed C1Failed C2Failed C3Passed C1–C3Removed after reviewUsed in our paper
BFCL12857533860146047
ToolSandbox1536100047040
τ²-bench1388916033016
AppWorld (new)446––3775669
Gaia2 (new)18029–01512149
Total321

Q3. Isolating the effect of candidate visibility

Following your suggestion, we added a comparison that keeps the scoring task and output format unchanged while varying whether the alternative candidate is shown. In both conditions, the judge scores only the target candidate, and we average two calls.

As reported in W2, showing the alternative improves performance for all three judges on both the selected Gaia2 groups and the selection groups. This supports a contribution from candidate visibility beyond the change from numerical scoring to direct choice.

Dear Reviewer MtY8,

Your review was really helpful in improving our draft. Since the deadline is approaching, could you be able to take a look at our responses and check if your concerns are resolved?

Thank you again for your time and effort in writing the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.