- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a concept called crossed instruction-trajectory groups. Basically, this looks at two possible actions that start off the same but split at one decision. The paper tests these actions under two instructions that differ in a single requirement, aiming for the “right” action to switch depending on the instruction. The idea is that a verifier only gets credit if it picks the correct action in both cases.
Critical strengths
Big strengths:
- The crossed design is a smart, clear way to check for fixed candidate biases. The three-level validation and the detailed error lists (Appendix G) make the experiment credible.
- Breaking down errors into types (fixed, position, tie, unresolved) reveals that similar scores can hide different problems. Controls—like withholding listings, matching output budgets, paraphrasing, and rerunning for noise—are thoughtfully done.
Critical weaknesses
But there are weak spots:
- The supporting data is limited and pretty clustered. Out of 195 groups, most are string matcher-friendly, and the main findings come from just 15 groups across five scenarios (eight of those sharing a key issue). Another 11 groups come from six listings, and there are no error bars on the main results. Some takeaways feel more like input formatting wins; the model can spot the cheapest choice because the table makes it obvious, not necessarily because of deep understanding. There's evidence that judges have trouble reading and comparing long JSON, which is different from the claim that "verifier preferences don’t follow the requirement.” Held-out experiments meant to mimic real-world use (Appendix R) show a mixed bag: some predictions don’t match up, the effects of different rendering choices blend together, and an important best-of-N comparison wasn’t run. Plus, there's no proof that better crossed accuracy leads to better real-world selection.
- Some results depend heavily on how scores are tallied. Ties automatically count as failures, so when using a three-level scale (-1, 0, +1) you get a lot of ties (like 51 for Qwen3.5-35B-A3B). This impacts key findings. Switching to mean aggregation brings back most of the lost signal, so a lot depends on the method, not the model. The way candidates are compared (one-by-one or side-by-side) changes what information is available; something the paper points out. Chance rates also move around (1/4 or 1/16) depending on how strict you are, making it hard to compare figures across the paper. The human audit leaves nine rows with no clear majority, and the agreement score is only 0.716, so some test cases (like list-index) look ambiguous. Overall, the main text feels jammed with detail and leans hard on the appendices.
Questions for authors
-Can you share per-cluster or bootstrap confidence intervals for the main results (like the 42-to-30 stat and the renderer recovery), treating each listing as a unique sample?
- Do crossed scores actually predict best-of-N or step-level performance in policy rollouts?
- How do your aggregation results change if you break ties by counting low-scored steps?
- If you shuffle the raw tables by item (instead of grouping), does the improvement persist, or does it drop off, basically, is this a true understanding effect or just nice formatting?
- If you drop the seven groups without a human audit majority, do your main conclusions hold? (Appendix F says yes, but please put this in the main text.)
7 comments
W1 & Q2. Connection to downstream selection
To test whether crossed failures provide useful feedback beyond our benchmark, we add a self-improvement experiment. We generate reflections from these failures, include them in the verifier’s prompt without updating its parameters, and evaluate its accuracy in judging agent steps on ToolPRMBench.
A Qwen3.8-27B model generates reflections for a Qwen3-4B verifier trained on AppWorld, BFCL, and ToolSandbox. In the no-feedback condition, reflections are generated from the verifier's own scores and answers on the training cases, without information about which answers are correct. The other conditions additionally identify cases that fail the instruction-only, trajectory-only, or crossed test. This comparison examines whether feedback from each test improves the usefulness of the generated reflections, rather than comparing the presence and absence of reflections.
We evaluate on 210 ToolPRMBench cases from tasks excluded from our training and development sets. This benchmark is relevant to downstream use: Li et al. [1] report that higher ToolPRMBench accuracy is associated with larger gains from best-of-8 search on GTA and BFCL. The table reports accuracy and the improvement from crossed feedback over each alternative, with 95% confidence intervals.
| Reflection from | ToolPRMBench accuracy (%) | Improvement from crossed feedback (percentage points) [95% CI] |
|---|---|---|
| No feedback | 48.5 | +4.2 [1.0, 7.5] |
| Instruction-only failures | 50.2 | +2.5 [−0.2, 5.5] |
| Trajectory-only failures | 49.0 | +3.7 [0.6, 6.8] |
| Crossed failures | 52.7 | — |
We also evaluate crossed accuracy on 57 held-out development groups:
| Crossed feedback compared with | Improvement in crossed accuracy (percentage points) [95% CI] |
|---|---|
| No feedback | +12.9 [6.4, 19.9] |
| Instruction-only feedback | +8.8 [1.2, 17.0] |
| Trajectory-only feedback | +9.9 [2.9, 17.5] |
On ToolPRMBench, crossed feedback improves accuracy over no feedback and trajectory-only feedback, with confidence intervals excluding zero. Its advantage over instruction-only feedback is less certain. On the development groups, crossed accuracy improves over all three alternatives.
These results show that feedback from crossed failures improves verifier accuracy beyond our constructed examples. The relationship between ToolPRMBench accuracy and search performance reported in [1] provides indirect evidence of practical relevance. We include these results in Section 4.4 and Figure 5, with confidence intervals in Appendix N, Tables 12-13.
[1] ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents.
W2 & Q3. Aggregation and the tie rule
We agree that this finding concerns how scores for individual steps are combined. Minimum aggregation assigns a trajectory its lowest step label (−1, 0, or +1). Two trajectories can therefore tie even when one contains more steps with low scores. Because unresolved ties count as failures, this can obscure differences between candidates.
Following the reviewer’s suggestion, we first compare the minimum labels. If they tie, we prefer the trajectory with fewer −1 steps, then fewer 0 steps if necessary. The table reports accuracy (%) with tie counts in parentheses for the BFCL setting highlighted by the reviewer and the newly added Gaia2 benchmark (also included in Appendix R.3, Table 26).
| Judge | Benchmark (groups) | Minimum | Then fewer −1 steps | Then fewer 0 steps | Mean |
|---|---|---|---|---|---|
| deepseek-v4.1-flash | BFCL value substitution (100) | 65 (35) | 76 (21) | 78 (13) | 78 (13) |
| deepseek-v4.1-flash | Gaia2 (149) | 32 (94) | 56 (43) | 64 (24) | 53 (33) |
| Qwen3.8-27B | BFCL value substitution (100) | 90 (8) | 93 (5) | 95 (2) | 94 (3) |
| Qwen3.8-27B | Gaia2 (149) | 22 (114) | 41 (70) | 48 (50) | 44 (53) |
| Qwen3.5-35B-A3B | BFCL value substitution (100) | 49 (51) | 62 (27) | 64 (24) | 64 (24) |
| Qwen3.5-35B-A3B | Gaia2 (149) | 22 (113) | 38 (72) | 46 (52) | 43 (57) |
For the specific case raised by the reviewer, Qwen3.5 on BFCL, this rule reduces ties from 51 to 24 and raises accuracy from 49% to 64%, matching mean aggregation. Across all combinations of judges and settings evaluated, the complete rule matches or exceeds mean accuracy, although some ties remain.
These results show that counting steps with low scores recovers distinctions lost when only the minimum is retained. We therefore narrow the claim to minimum aggregation with coarse labels and unresolved ties, rather than a limitation of evaluating individual steps in general.
Thank you for the detailed review and concrete suggestions. We have expanded the evaluation beyond the groups used in our diagnosis, added controls for how information is displayed and scores are produced, and tested whether feedback from crossed failures improves performance on an external benchmark. We also clarify the scope of our conclusions below.
W1 & Q1. Additional evaluation and uncertainty across scenario clusters
We test whether the failures observed in the original 15 selection groups also occur in 29 additional groups: 11 held-out selection groups and 18 AppWorld groups. We also remove the two largest scenario clusters to check whether a few heavily represented scenarios drive the result.
Each group is evaluated by five judges. The table counts judgments that are correct under the original instruction and those correct under both instructions; their difference is the additional failures exposed. Counts are summed across judges, so 15 groups yield 75 judgments. Confidence intervals are computed by resampling scenario clusters because multiple groups can share the same scenario or tool listing.
The counts use the first pairwise call for each judge and group. Confidence intervals come from 5,000 bootstrap resamples of the scenario clusters.
Additional failures persist in groups not used in diagnosis and after removing the two largest clusters, including the shared reservation highlighted by the reviewer. The wide interval for the original set reflects its small number of clusters. The added groups strengthen the evidence, although the limited number of clusters still affects precision.
We compare the original tool output with two alternative representations on 11 held-out selection groups. The raw table organizes observed values by item without computing the selection attributes, while the computed table additionally provides derived attributes used for selection. We estimate gains over the original output using paired bootstrap resampling of scenario clusters.
Gains are percentage-point differences from the original output. Both table representations improve performance for all five judges, with confidence intervals excluding zero. The raw-table gains show that recovery does not require supplying precomputed selection attributes. The additional-group analysis appears in Section 5.1, Table 4, and the renderer confidence intervals are reported in Appendix P, Table 14.