acceptodds
Under review as a conference paper at ICLR 2027

RePair: Rubric-Guided Turn-Level Preference Learning for Tool-Using Agents

Abstract

Tool-using agents on long-horizon tasks keep making the same decision errors, and the two standard remedies miss them for different reasons. Supervised fine-tuning imitates expert trajectories, which never visit the states the learner reaches after its own mistakes. Reinforcement learning from a terminal reward does visit those states, but it pays for full rollouts and still cannot say which turn went wrong. We introduce RePair (Rubric-guided Repair as preference Pairs), which turns the learner's recurring errors into a small set of reusable behavioral rubrics. A rubric names an observable error pattern, states the rule that corrects it, and gives a criterion that a repaired turn must meet. Whenever a rubric matches a failed turn, a repair model rewrites the turn as the rule prescribes, and a validator checks the rewrite against the criterion using only what the agent could see at that turn, so no rollout is needed. The rewrite and the learner's original response then form a preference pair on the very same input, and since that input carries no rubric, the trained policy needs none at inference. On BrowseComp-Plus, rubrics induced automatically from the learner's own failures lift a supervised Qwen3-14B search agent from 42.6% to 53.2% with 2,097 pairs. The learner can also write its own repairs: with no teacher supplying the preferred responses, two rounds of self-repair still gain 7.2 points over the same baseline, and the second round adds to the first, so an agent can keep improving from its own failures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, tool use, direct preference optimization, failure-driven learning, rubric induction, self-improvement, deep research

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kmxROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RePair, a rubric-guided framework that converts recurring errors in failed agent trajectories into locally validated preference pairs for offline training, without additional rollouts. Experiments on BrowseComp-Plus show that RePair improves Qwen3-14B accuracy from 42.6% to 53.2% with only 2,097 pairs, and supports iterative self-repair.

Critical strengths

The paper addresses an interesting and important problem with a well-motivated and natural approach. The proposed method achieves strong empirical results and provides several interesting insights into improving tool-using agents.

Critical weaknesses
  1. The experimental evaluation could be strengthened by comparing RePair with more existing methods.
  2. The experiments are conducted only on BrowseComp-Plus. Evaluating RePair on more diverse tasks and environments would help demonstrate its generalizability.
  3. The contribution of rubric-based validation is not sufficiently explored. The current no-rubric baseline removes both rubric-guided repair and rubric-based validation, making it difficult to isolate their individual contributions.
  4. The presentation could be improved, particularly Figures.
Questions for authors

See Weaknesses. If the authors can address these concerns through further clarification or additional experiments, I would be willing to raise my score accordingly.

1 comment
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer kmxR,

Thank you again for your constructive feedback. With less than four hours remaining before the deadline, we wanted to kindly follow up on our responses and the revised manuscript.

In response to your concerns, we added evaluations on four held-out multi-hop QA benchmarks and a second model family, an ablation that removes only the LLM validator while retaining rubric-guided repair, and a discussion of published agent-RL results with explicit caveats about differences in evaluation settings. These additions are detailed in our responses above.

We would greatly appreciate any feedback on whether these additions address your main concerns, or whether there are remaining issues we could clarify before the deadline. We understand that this is a busy period and would be grateful even for a brief update.

Thank you for your time and consideration.

Reviewer_DWdrOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RePair (Rubric-guided Repair as preference Pairs), a post-training method for tool-using agents on long-horizon tasks. Instead of imitating expert trajectories (which never visit the learner's post-mistake states) or RL with terminal rewards (which pays for full rollouts and gives weak credit assignment), RePair converts the learner's recurring errors into reusable behavioral rubrics. Each rubric is a triple (trigger γ, hint η, criterion κ): the trigger names an observable error pattern, the hint states a general corrective rule, and the criterion checks a repair using only information available at that turn. Rubrics are induced automatically from failed rollouts via a diagnosis–clustering–consolidation–reassignment pipeline. A repair model rewrites a flagged failed turn under the rubric's guidance; a validator checks the rewrite locally (no rollouts). The accepted repair and the original failed response form a turn-level DPO pair on the same input, which contains no rubric — so the trained policy needs no rubric at inference. On BrowseComp-Plus, an induced codebook (25 rubrics) lifts a supervised Qwen3-14B agent from 42.6% to 53.2% mean@3 with 2,097 pairs, beating a hand-written codebook; two rounds of self-repair (learner repairing its own failures) gain 7.2 points.

Critical strengths

Clear, well-motivated problem positioning. The paper cleanly articulates why SFT (expert states, Ross et al. 2011) and outcome-reward RL (rollout cost, weak turn-level credit) both miss recurring decision errors in long-horizon agents, and carves out a genuinely distinct third route: reusable, inspectable corrective rules used at training time only, then distilled into weights. Well-executed ablation battery. Section 4.1 is unusually thorough: no-rubric repair (same teacher, same failures, no rubric) recovers only 3.41/10.64 points; full-trajectory SFT on 7× more teacher turns reaches 48.19; chosen-only SFT 49.20 (and collapses reasoning length); rubrics-in-prompt 40.96 vs. 53.21 when trained. These ablations genuinely isolate that the rubric, not the teacher or the pair machinery, carries the gain. Automatic induction actually works (RQ2). The induced 25-rubric codebook beats the 3-rubric hand-written one (53.35±0.62 vs. 49.40±1.32 across three seeds, Welch's t-test p=0.02) with 44% fewer pairs — the paper honestly notes this is not a controlled comparison (25 vs. 3 rubrics confound coverage with pair count), which is appropriate caution. Self-repair improving across rounds (RQ3) is a notable finding. Two self-repair rounds recover 7.2/10.6 points with no external teacher supplying preferred responses, and round 2 improves on round 1 (50.14±1.14 vs. 46.25±0.95, p=0.01) — evidence the agent can keep improving from its own failures under a fixed codebook and validator. Honest cost accounting and strong reproducibility. The token-cost table (67.4M vs. an estimated 957M for 128 GRPO steps) is presented with explicit caveats (not accuracy-matched; most RePair tokens come from larger models). Code, data, all three codebooks, and all prompts are released at an anonymous repository; hyperparameters were fixed before test evaluation; three training seeds plus paired bootstrap are reported. Careful design details. Local validation using only turn-visible information, specificity checks preventing question-dependent rules from leaking the answer, and the validator-human agreement study (88.7%, κ=0.76 on 150 candidates) show real attention to the soundness of the data pipeline.

Critical weaknesses

Single-benchmark, single-family evidence. All results come from one environment (BrowseComp-Plus with three tools: search / get_document / finish), one model family (Qwen3-14B/8B), and one warm start. The paper concedes this in Limitations, but for ICLR the generalization question is central: nothing shows rubrics induction works in environments with more heterogeneous tools, non-retrieval actions, or non-binary outcome signals. The 8B induced codebook (14 rubrics) is a second data point but from the same family and setting. Pipeline depends on many strong external models; sensitivity uncharacterized. Diagnosis/repair use kimi-k3, consolidation gpt-5.6-sol, validation deepseek-v4-flash, judging gpt-oss-120b, the reader deepseek-v4-flash. The final gain plausibly depends on these choices (e.g., a weaker repair model or validator), yet there is no experiment varying any of them. The paper itself flags that induced rubrics "inherit some of that model's uncertainty" — no analysis of codebook stability across induction runs, inducing models, or failure samples is provided. This matters for the central claim that rubric induction is a reliable recipe. Local validation is a proxy whose long-run effect is unquantified. Acceptance is explicitly "a local judgment rather than evidence of eventual task success." How often does an accepted repair actually lead to task success if rolled out? A counterfactual check (roll forward a sample of accepted repairs) would quantify the noise floor of the preference signal; label smoothing (ε=0.1) partially hedges this but is never ablated. Relatedly, ~11% validator–human disagreement implies non-trivial label noise at scale. Self-repair needs 4 rollouts per question and ~3× pairs. The "no teacher" claim is appealing but the external infrastructure (judge, labeler, validator) remains; the honest framing would be "no external repair model," which the paper does state, but readers may over-credit RQ3. Also self-repair reaches only 49.80 vs. 53.21 teacher repair. Some comparisons are unequal in compute/budget. GRPO is compared by token count against an estimated, not implemented, recipe (Pan et al. 2026, 16×8 rollouts × 128 steps) — the paper caveats this, but a small actual RL run (even 32 steps) as an empirical anchor would be far more convincing than an estimate; as is, the headline "rollout-light" claim rests partly on extrapolation. Missing uncertainty for most ablation arms. Three seeds exist for the two 14B codebook arms and self-repair rounds, but Table 3's single-seed gaps (e.g., chosen-only SFT 49.20 vs. RePair 53.21, gap 4.01) have no variance attached; some conclusions in §4.1/§4.2 (e.g., Top-5 47.79 < Top-1 48.80) may be within noise, and the non-monotone top-k result is not discussed. Questions left open about the state rendering assumption. The harness re-renders input from a compact state, enabling single-turn repair. It is unclear how much of the method's cost advantage stems from this custom harness rather than the rubric method itself, or whether the approach transfers to agents with raw transcript contexts where s_t cannot be cheaply replayed.

Questions for authors
  1. If a sample of accepted repairs were rolled forward to completion, what fraction would produce correct final answers? This would quantify how well local acceptance proxies eventual success.
  2. How stable is the induced codebook across (a) different failure-pool samples, (b) different inducing models, (c) repeated induction runs? The Limitations section names this question but the paper does not answer it.
  3. What happens with a weaker repair model or validator (e.g., the learner itself as repair model is tested, but a weaker validator is not)? Is the method's gain robust to validator quality, or does the 88.7% agreement rate need to be ≥95%?
  4. Is ε (label smoothing) ablated anywhere? Given that acceptance is a validator judgment, the choice ε=0.1 seems important but unexamined.
  5. For the top-k subset experiment (Table 4), Top-5 (47.79) underperforms Top-1 (48.80) despite more pairs — is this within seed noise, or does adding certain rubrics hurt? Please discuss.
  6. How does RePair interact with on-policy RL? E.g., does a RePair-initialized policy trained further with GRPO converge faster or to a higher final accuracy than GRPO alone?
  7. How much of the token-cost advantage comes from the compact-state harness (re-renderable s_t) rather than the rubric machinery? Would the method work with raw transcript contexts?
  8. Round-2 self-repair: do failure modes shift across rounds (i.e., does the codebook keep matching as the policy improves), or would round 3+ require codebook re-induction? Is there any evidence on a third round?
Reviewer_HoMwOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RePair, a post-training method for making more robust post-training datasets by expanding coverage of situations where the learner fails and enters states not encountered in expert trajectories. The method consists of taking a failed trajectory, running a codebook of rubrics on it to find errors, and generating a new version of the trajectory from the point of failure with a repair model or the learner itself. New trajectories are sampled until one is deemed acceptable by the judge with the codebook. The original failure and the accepted trajectory then form a preference pair and the learner is fine-tuned with DPO on generated preference pairs. The paper also presents a method for inducing codebooks by applying a prompting rubric-induction framework to failed trajectories in an existing dataset. The overall contributions are considered 1) the full codebook induction + preference pair generation + fine-tuning method and 2) a version where the learner, rather than an expert teacher, generates the repaired trajectory.

Experiments are conducted in two settings: Qwen3-14B and Qwen3-8B, both with/on BrowseComp-Plus. Results are discussed primarily for 14B. The paper asks three research questions: 1) does repair help with a handwritten codebook? 2) can the codebook be induced automatically from the learner's own failures (and improve over the handwritten codebook)? 3) can the learner supply the repairs? The paper concludes that the answers are yes, yes, and yes because in all three cases, the Qwen3-14B 42% mean@3 baseline is improved by several percentage points. There is ablation analysis: no rubrics, SFT only, pairs generated but only accepted pair member used and for SFT, rubrics in prompt rather than as part of the training data pipeline. All underperform the full RePair method. The paper positions itself against general agentic post-training and rubrics for training. It also acknowledges the limited experimentation.

Critical strengths

Strength 1: this paper is very well-written - I found it easy to understand the majority of the time, and when I was confused the answer usually came soon. The idea is clean and well-presented. The results structure follows clearly from the claims, the analysis is presented clearly, and the whole paper is laid out well.

Strength 2: the experiment we do see is investigated deeply and shows at least a good instance of the method - the induced rubric seems high-quality and performance gain is significant. The ablations are well-designed and tell us a lot about the method's value and mechanics, and this is all realized and discussed in the analysis section.

I think this is an interesting line of work that with more experimentation behind it can become conference-ready.

Critical weaknesses

The primary weakness is that the paper does not have enough experimentation. It has only two settings, one trial in each, and only one of those is discussed deeply. This has many limitations:

  • It's not enough to know much about the method. A few different models should be tried. Furthermore, given how many steps there are in the pipeline that involve stochasticity, more trials would be very important. I realize experimentation can be expensive but right now the value of the method is not empirically proven.

The next weakness, which Is connected to the first, is that the paper makes an unsound comparison between the hand-written codebook (three rubrics written from reading an unspecified number of failed trajectories) to the induced codebook (25 rubrics induced from 187 trajectories). The research questions are separated into handwritten and induced codebooks, I think in part to separate the training and the codebook, which I think is a good idea. But the induced codebook has eight times as many rubrics and we can see that they legitimately cover far more ground than the handwritten rubrics. The comparison is therefore completely unfair. The paper does acknowledge that it's not a perfect comparison but it claims that it's useful enough to answer RQ2. I agree that it answers that an induced codebook can be used without harm, but it's not a fair comparison.

This alone might seem like a nitpick especially since RQ2 was about whether an induced codebook is viable, not whether it's better, but the whole thing is very strange to include. A handwritten codebook with three rubrics just doesn't seem like a setting that tells us anything; furthermore, the fact that performance didn't increase that much with a codebook eight times the size of the handwritten one suggests that the training may dominate in importance. A baseline/ablation with an expanded DPO dataset that wasn't repair-specific, or RL, might tell us more (and also help us better understand RePair's position on the cost-performance curve). This, on top of the fact that the paper is a bit repetitive and filler-heavy (see below) and only has one experimental setting, makes the overall work seem half-baked.

The paper is also somewhat repetitive, possibly (though I'm not sure) to fill space given the lack of material from the limited experimentation. In section 2, the intro and subsection 2.1 are somewhat filler-like. Subsection 2.4 is unnecessary. Subsection 2.3, though necessary, is longer than it needs to be - it's an expected field-standard structure.

Questions for authors
  • The accepted trajectory may not be a success, it may just be accepted by the LLM judge because it solves the error locally around the point of error. I was intrigued by this - it seems like it has a benefit of promoting exploration and coverage and not just collapsing to what the teacher is good at or a few successful strategies. Is this true?
  • How did you validate intermediately - e.g. that your prompts were working well and finding good failure points? Though I can see that the method works end to end I would be curious about this.
  • The contributions roll the entire separate-repair-model method into one, then have self-repair as a separate contribution (though to be fair, not a separate method). Given that the codebook and the training are 1) modular and 2) separated out in the research questions, I wonder if it would be valuable to separate them in the contributions. I remember reading it and thinking the contribution structure didn't flow well with the way the intro had described the method, which was very two-part.
  • I realize that the method is rollout-light because full trajectories are not regenerated. But I wonder

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.