- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MIRA mines workflow scaffolds and policy constraints from ABCD dialogues, synthesizes customer-service scenarios, and grades full agent transcripts for task completion, tool correctness, and policy adherence. It evaluates five agents over repeated simulated conversations and uses multiple LLM judges plus a double-annotated human sample. A separate regrading of released τ²-bench transcripts is used to argue that endpoint-oriented evaluation misses behavioral policy violations.
Critical strengths
I think the distinction between completing a task and following the correct procedures is important. An agent may reach the right final result but still skip required steps, such as verification or user confirmation. These failures may not be captured by evaluations that only check the final outcome. Re-evaluating the same transcripts is a reasonable way to study this problem.
The authors also put considerable effort into validating their evaluation method. They use multiple judges from different model providers, evaluate individual constraints, and include 190 transcripts annotated by two human reviewers. These efforts make the evaluation more convincing than simply relying on a single LLM judge. I also appreciate that the authors report the differences in scores across judges rather than hiding them.
Critical weaknesses
The core problem is that the paper has limited theoretical justification and lacks sufficient comparison with existing methods. While the paper compares MIRA with τ²-bench, it does not directly compare against related methods. This means the paper shows that MIRA can detect some failures missed by traditional evaluations which is a known problem, but it does not clearly demonstrate that MIRA performs better than existing automated evaluation methods.
The central baseline and examples do not currently establish valid measurement of policy violations, as the traces themselves are not necessarily the only correct path. Also, the paper repeatedly describes τ²-bench as reporting 100% policy adherence, but §3 and Appendix C acknowledge that this is the authors’ own re-derived quantity, not a published policy-adherence submetric. More seriously, Appendix M’s supposedly unambiguous examples include policy interpretations that conflict with the official retail policy currently available: pending orders may have their payment method changed, returns may be refunded to an existing gift card, and item modifications are explicitly collected into one call. This does not prove those particular trajectories are compliant, because other conditions or a different policy revision may apply; however, the stated reasons are insufficient and undermine the examples offered to validate the judge. The exact benchmark revision, scoring derivation, applicable clauses, and independently adjudicated traces are essential before the headline gap can be trusted.
Several central conclusions overstate what the experimental design identifies. All three MIRA dimensions are judge-derived under stubbed tools, yet the 27.1% conditional result is described as passing end-state and action checks (§6.1; Appendix D). The τ²-to-MIRA comparison also changes the agent cohort and the grading mode from holistic to checklist, so it does not isolate mining. Finally, Figure 2 labels its first statistic “single trial” although the caption defines it as success at least once in three trials, and concrete user requests are regenerated across trials. These results cannot establish the claimed single-trial overestimation or agent-only inconsistency without separating sampling definitions, user variation, and attainability under the six-turn cap.
Questions for authors
For the reported 27.1% failure rate, were any actual environment-based checks used to verify the final state and actions? If all three evaluation results come from LLM judges, I think the paper should make this clearer. It would also be helpful to independently validate these cases, especially those where human annotators disagree.
Could the authors report the actual success rate for a single trial, as well as the rate of succeeding in all three trials? I would also like to see whether allowing more conversation turns changes the results, preferably using the same user goals across trials. Finally, for the mining ablation, could the authors keep the policy information and grading method consistent, and directly evaluate policy adherence rather than only task completion?
you were right about Appendix M, and it led us to re-check all 127 failing cases against the policy text clause by clause — none of them fails on a reason the policy contradicts, so the 72.1% result stands unchanged.
One correction first, because you found a real error and we would rather state it plainly.
You are right about Appendix M: a pending order's payment method can be changed, a return refund can go to an existing gift card, and putting all item changes in one call is required rather than forbidden. Looking into why, we found something worse that we will also report — three of those four examples came from an earlier judging run by different judges, not the judge whose scores the paper reports. Under the reported judge all three pass, scoring 0.85 to 1.00.
So rather than swap in better examples, we re-checked everything: we pinned the policy text and went through all 127 conversations the reported judge marked as failing policy, clause by clause. None fails on a reason the policy contradicts, and your three mistaken readings appear zero times. The judge in fact draws a finer line than the objection — it limits refunds to the original payment method only for cancellations, where the policy does say that, and allows a gift card for returns, where the policy allows it. The headline was always computed from the reported judge and is unchanged: 72.1% is 329 of 456. The revision will replace the examples with ones from those 127, cite each to a policy line, and state the correction openly.
Q1. For the 27.1% figure, were any actual environment-based checks used to verify the final state and actions?
If all three come from LLM judges, the paper should say so. It would also help to independently validate these cases, especially where annotators disagree. No. Tool responses are fixed synthetic values, not a live system, so there is no final database to assert against and no reference action trace to compare to — all three scores come from the judge reading the conversation, and the revision will say so directly. The result should be read as one judge disagreeing with itself across three dimensions on the same conversation. We think that is still worth reporting, because the three things come apart under one grader, but it does not show a deterministic benchmark would have passed those runs and we will stop implying it. The figure also becomes 26.8% (60 of 224) once the failed judge call is removed — the same thing behind the 149 you flagged. On scope we should be straightforward: we compare against τ²-bench and against a change of grader on fixed conversations, which shows an end-state check leaves policy adherence unmeasured. It does not show MIRA beats other automated evaluators, and the revision will not suggest it does.
Q2. Could you report the actual single-trial success rate and the all-three-trials rate, say whether more turns changes the results using the same user goals, and for the ablation keep policy and grading consistent while directly evaluating policy adherence?
Scoring each trial on its own, the rate at which a model passes all three dimensions is 8.0%, 6.9% and 6.9% (60, 52 and 52 of 750), against pass¹ up to 24.0% and pass³ never above 2.7%. You were also right that the panel labels contradicted the caption; they will read "≥1 of 3 / ≥2 of 3 / all 3 trials", with the per-trial rates alongside. Only one of the two gaps is a reliability claim: going from "at least once" to "every time" costs most of the measured performance, while a single trial sitting below "at least once" is arithmetic. On turn budget we have not run it and will not guess. Every conversation runs to the six-turn limit, so a false promise in a final turn could be truncation; separating that from agent and user-simulator variation needs a re-run with the user turns held fixed, which we will list as a limitation.
On the ablation, the hand-authored scenarios are already re-scored with the current checklist for the three shared models, leaving one judge-matched pass of about 30 minutes. We will post it in this period. We will say in advance that the ablation we already have, on task completion, works against our own mining claim — hand-authored templates separate models more widely, 45.3 against 22.7 points. You were also right that the comparison changes cohort and grading mode together, and we will label it a combined effect. One correction to the direction, our fault for never writing it down: on MIRA the judge marks each mined rule pass or fail and the rubric folds those into three values, while τ² has no per-task rules so the judge gives one score taking 21 values. The checklist side is the coarser one at the threshold.
Given that the headline result survives a clause-level re-check of all 127 failing cases, that the provider-disjoint judge makes the finding stronger rather than weaker, and that the specific errors you identified will be corrected openly in the revision, we hope you would consider revisiting the soundness score.