- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the conflation of plan deviation and task failure in agent judges. It proposes DecoupledJudge, which uses isolated inputs and two separate LoRA adapters to assess plan adherence and task correctness. Experiments on tau-bench retail and airline trajectories report reduced conflation and improved both-label accuracy, with ablations, scaling experiments, and cross-domain evaluation.
Critical strengths
Distinguishing plan adherence from task correctness is useful, particularly for executions that deviate from a plan but still succeed. The method is straightforward, and reporting both conflation and both-label accuracy avoids treating different predictions alone as successful disentanglement. The released predictions also allow the main-table metrics to be recomputed.
Critical weaknesses
-
The experimental evidence is insufficient. The main open backbones, Qwen2.5 and Llama-3.1, are dated, and comparisons with adapted versions of representative judges are missing, including JudgeLM (ICLR 2025), Prometheus 2 (EMNLP 2024), and Agent-as-a-Judge (ICML 2025). In the supplied data, 51/121 conflict test traces share underlying tasks with training, and the joint and separate DEV judges receive unequal supervision (185 vs. 1,052 trajectories). API comparisons also use different test subsets. These issues weaken the causal and general conclusions.
-
The evaluation target and practical benefit need stronger justification. Official tau-bench r_outputs checks required strings in responses, rather than final database correctness, so whether VIOL measures full task success is unclear. The paper also does not establish what separate DEV prediction adds beyond outcome evaluation, particularly when DEV itself is generated by deterministic rules.
Questions for authors
What practical benefits does separately assessing plan adherence and task correctness provide over evaluating task outcomes alone, and are there experiments demonstrating these benefits?
Response
I haven’t received the review response, could this be an issue on my end?