acceptodds
Under review as a conference paper at ICLR 2027

Deviation Is Not Failure: Decoupling Plan-Adherence from Task-Correctness in LLM Judges via Input-Isolated Single-Axis Fine-Tuning

Abstract

As agents execute longer tool-rich workflows, evaluating them increasingly relies on an LLM judge that inspects the execution trace. Such a judge is implicitly asked two orthogonal questions: did the agent deviate from its stated plan (a procedural question, DEV), and did the outcome violate the task objective (a correctness question, VIOL)? We first show that state-of-the-art judges systematically conflate these two axes: on tau-bench traces whose two gold labels disagree, even the strongest API judge with a carefully decoupled, evidence-forced prompt returns the same verdict for both axes 87–90% of the time and answers both correctly only 9–12% of the time–and an explicit "decouple-then-judge" prompt is worse than a joint one. We trace the cause to a shared input: a single judge that sees the plan, the spec, and the trace at once cannot help but let procedure color its correctness verdict. DecoupledJudge turns this diagnosis into a method, training two single-axis judges whose inputs are isolated by construction–the DEV judge sees only the plan and the trace, the VIOL judge sees only the spec, the trace, and the delivered outputs–so neither axis can copy the other. On the orthogonal-conflict subset, DecoupledJudge cuts conflation from 0.71 (untuned base) and 0.54 (best API prompt) to 0.16, and raises the fraction of traces on which both axes are correct from 0.17 to 0.84, with paired bootstrap CIs excluding zero and an exact McNemar test at p<10^-12. The decoupling transfers under leave-one-domain-out, a controlled ablation shows the input isolation itself (not the training data) is causally responsible, and the effect is scale-invariant across 3B, 7B, and 14B, confirming that conflation is an architectural artifact of shared inputs rather than a small-model deficiency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM-as-a-judge, agent evaluation, process supervision, tool-use agents, input isolation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NXtdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the conflation of plan deviation and task failure in agent judges. It proposes DecoupledJudge, which uses isolated inputs and two separate LoRA adapters to assess plan adherence and task correctness. Experiments on tau-bench retail and airline trajectories report reduced conflation and improved both-label accuracy, with ablations, scaling experiments, and cross-domain evaluation.

Critical strengths

Distinguishing plan adherence from task correctness is useful, particularly for executions that deviate from a plan but still succeed. The method is straightforward, and reporting both conflation and both-label accuracy avoids treating different predictions alone as successful disentanglement. The released predictions also allow the main-table metrics to be recomputed.

Critical weaknesses
  1. The experimental evidence is insufficient. The main open backbones, Qwen2.5 and Llama-3.1, are dated, and comparisons with adapted versions of representative judges are missing, including JudgeLM (ICLR 2025), Prometheus 2 (EMNLP 2024), and Agent-as-a-Judge (ICML 2025). In the supplied data, 51/121 conflict test traces share underlying tasks with training, and the joint and separate DEV judges receive unequal supervision (185 vs. 1,052 trajectories). API comparisons also use different test subsets. These issues weaken the causal and general conclusions.

  2. The evaluation target and practical benefit need stronger justification. Official tau-bench r_outputs checks required strings in responses, rather than final database correctness, so whether VIOL measures full task success is unclear. The paper also does not establish what separate DEV prediction adds beyond outcome evaluation, particularly when DEV itself is generated by deterministic rules.

Questions for authors

What practical benefits does separately assessing plan adherence and task correctness provide over evaluating task outcomes alone, and are there experiments demonstrating these benefits?

Reviewer_cd2pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLM judges confuse an agent's departure from its plan with failure to complete the task. DecoupledJudge uses two LoRA adapters: one judges deviation from the plan and trace, while the other judges correctness from the specification, trace and output, without seeing the plan. The labels come from a write-operation checker and the environment's output assertions, respectively.

On 121 τ-bench trajectories where the gold labels disagree, the 7B model reduces conflation to 0.16 and gets both labels right on 84% of cases. The experiments compare prompted judges, isolate the effects of inputs and adapters, and test different model sizes and cross-domain transfer.

Critical strengths

Separating plan adherence from task success is useful, and the conflict subset exposes errors that aggregate accuracy would miss. Deriving the two labels from separate programs also makes the diagnostic easier to interpret. I appreciate the controlled input/adapter ablations, paired tests and additional model-family experiment; these provide more evidence than a simple before-and-after comparison.

Critical weaknesses

My main concern is the explanation of the improvement. Joint FT already reaches VIOL accuracy 0.83, close to DecoupledJudge's 0.85, while still seeing the plan. The results therefore do not establish that conflation is an unavoidable architectural problem. No API baseline receives exactly the same isolated inputs, so a straightforward alternative remains untested: two prompted calls, with the plan removed from the correctness call. It is also unclear what the learned DEV judge adds when its labels come from a deterministic checker that could be used directly.

The broader robustness claims need qualification. The 7B conflation rates are 0.157, 0.339 and 0.331 across seeds, averaging 0.276 rather than the headline 0.16. Held-out-domain VIOL accuracy falls from 0.85 to 0.53, and the airline conflict subset contains only eight cases. There is a further issue with observability: some correctness labels depend on database facts absent from the trace, and human agreement for VIOL is low. Replaying the oracle establishes deterministic labels, not that a trace-only judge can recover them. Without a clearer account of these cases and how the conflict trajectories were generated, I would interpret the results as a useful in-domain fix rather than a general explanation of judge failure.

Questions for authors

How do API judges perform with exactly the same isolated inputs? This would most affect my assessment of whether the improvement requires fine-tuning. Please also report seed-averaged comparisons and distinguish results on the 48- and 121-trace subsets.

How were the plans obtained, and what fraction of the conflict trajectories arose naturally? Can you separate VIOL cases verifiable from the supplied trace from those requiring hidden database state? Please also reconcile the conflicting L1/L3 numbers.

2 comments
Reviewer_cd2pOpenReview ↗

Response

I haven’t received the review response, could this be an issue on my end?

Area_Chair_o1osOpenReview ↗

authors not required to respond

Authors are not required to respond to reviewers. It is fairly typical for authors to not respond to reviews when they don't feel they have a chance of changing the reviewer's mind about the paper.

-AC

Reviewer_LR3VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a failure mode in LLM-based trajectory evaluation: judges often conflate whether an agent deviates from its declared plan (DEV) with whether the agent violates the task objective (VIOL). The authors introduce DecoupledJudge, which trains two single-axis LoRA adapters with isolated inputs: the DEV judge receives the plan and execution trace, while the VIOL judge receives the task specification, trace, and delivered output but not the plan. On a selected subset of τ-bench where the two gold labels disagree, the proposed method reduces the conflation rate from 0.71 to 0.16 and increases the fraction of examples on which both axes are correct from 0.17 to 0.84. The paper evaluates joint fine-tuning, shared and independent adapters, multiple model sizes, a second backbone, prompt-based API judges, quadrant ablations, and leave-one-domain-out transfer.

Critical strengths
  1. The paper identifies a practically important evaluation failure mode. Plan deviation and task failure are conceptually distinct, and conflating them can incorrectly penalize successful recovery, replanning, and alternative solution paths. The deviant-but-correct and faithful-but-failing cases provide a useful diagnostic setting that would be obscured by aggregate accuracy on naturally correlated data.

  2. The empirical conflation effect appears across several model families and prompting configurations. The proposed remedy is also simple and inexpensive to reproduce, and the paper includes useful comparisons with joint fine-tuning, a shared adapter, different model sizes, and leave-one-domain-out evaluation.

Critical weaknesses
  1. The VIOL task may not be identifiable from the judge’s input. The gold label is produced using database-level output assertions, while the VIOL judge observes only the specification, trajectory, and delivered output. The paper does not establish that these observations expose all environment state required to determine the oracle label. This concern is reflected in the human audit, where VIOL agreement is only 0.69 with Cohen’s κ of 0.38, and in leave-one-domain-out evaluation, where VIOL accuracy drops from 0.85 to 0.53. The reported in-domain accuracy may therefore reflect task templates or domain-specific trajectory shortcuts rather than verification of task correctness. This directly undermines the interpretation of the main result.

  2. The experiments do not establish that input isolation is the causal mechanism claimed by the paper. The complete method simultaneously isolates the input, separates the tasks, introduces independent adapters, removes gradient competition, and increases task-specific parameter capacity. In Table 4, input isolation reduces conflation from 0.49 to 0.35, while changing from a shared adapter to independent adapters further reduces it from 0.35 to 0.16. A necessary control is two independently trained adapters that both receive the complete input. Moreover, the DEV label is generated through a deterministic comparison of declared and executed write operations, but no rule-based parser/diff baseline is included. The headline exact-match improvement is driven primarily by DEV accuracy increasing from 0.81 to 0.98, while VIOL accuracy improves only from 0.83 to 0.85 relative to joint fine-tuning. The current evidence therefore does not support the paper’s stronger claim that input isolation itself solves correctness conflation.

Questions for authors
  1. Does the VIOL judge observe the complete final environment state required to evaluate every database-level output assertion? If not, how is the VIOL label identifiable from the model input? Could the authors evaluate counterfactual pairs in which the task and trajectory pattern remain similar but the final environment state changes?

  2. What is the performance of two independently trained, capacity-matched adapters when both receive the complete input? This control appears necessary to separate the effect of input isolation from parameter isolation and task specialization.

  3. What performance is obtained by directly applying the deterministic plan–trace comparison used to construct the DEV labels? If this rule-based evaluator is nearly perfect, what deployment setting requires an LLM-based DEV judge?

  4. From joint fine-tuning to DecoupledJudge, VIOL accuracy changes only from 0.83 to 0.85, while DEV accuracy changes from 0.81 to 0.98. Can the authors show that input isolation materially improves correctness evaluation itself, rather than primarily improving reproduction of the DEV labeling rule?

The answers to Questions 1–3 would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.