acceptodds
Under review as a conference paper at ICLR 2027

MIRA: Mined Interaction Replay for Agent-Evaluation–Measuring Policy Adherence Beyond the End State

Abstract

Benchmarks for tool-augmented conversational agents check task completion against the final environment state and tool correctness against the action trace. Policy adherence—verifying identity before acting, honoring a required confirmation, not claiming actions never taken—is not checked directly but proxied from the same end-state check. We show this proxy is unsound. Regrading -bench's own released transcripts on behavior, with agents, scenarios and tool traces held fixed, drops all four of its agents from the 100% policy adherence its native grader reports to 69.7%–76.3%: the proxy and the behavior disagree within the same run. We introduce MIRA (Mined Interaction Replay for Agent-Evaluation), which measures policy adherence directly. MIRA mines workflow graphs and ordering policies from real customer-service dialogues, synthesizes multi-turn scenarios from them, and scores each full transcript—tool calls, ordering and policy compliance along the path taken—with an LLM judge validated against double-annotated human labels, alongside a provider-disjoint second judge. Across conversations with five frontier models, policy adherence averages just 28.9% and spans 12.2%–40.9% across models, and 27.1% of runs that pass both the task-completion and tool-correctness checks still violate policy. The model ordering is identical under all three frontier judges. Reliability is also worse than single-trial scoring suggests: no model passes the same scenario in all three trials more than 2.7% of the time.

Primary area:
datasets and benchmarks
Keywords:
valuation methodologies, benchmarking, NLP datasets, metrics, automatic evaluation of datasets, Multi-turn conversation evaluation, agentic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UWGdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MIRA mines workflow scaffolds and policy constraints from ABCD dialogues, synthesizes customer-service scenarios, and grades full agent transcripts for task completion, tool correctness, and policy adherence. It evaluates five agents over repeated simulated conversations and uses multiple LLM judges plus a double-annotated human sample. A separate regrading of released τ²-bench transcripts is used to argue that endpoint-oriented evaluation misses behavioral policy violations.

Critical strengths

I think the distinction between completing a task and following the correct procedures is important. An agent may reach the right final result but still skip required steps, such as verification or user confirmation. These failures may not be captured by evaluations that only check the final outcome. Re-evaluating the same transcripts is a reasonable way to study this problem.

The authors also put considerable effort into validating their evaluation method. They use multiple judges from different model providers, evaluate individual constraints, and include 190 transcripts annotated by two human reviewers. These efforts make the evaluation more convincing than simply relying on a single LLM judge. I also appreciate that the authors report the differences in scores across judges rather than hiding them.

Critical weaknesses

The core problem is that the paper has limited theoretical justification and lacks sufficient comparison with existing methods. While the paper compares MIRA with τ²-bench, it does not directly compare against related methods. This means the paper shows that MIRA can detect some failures missed by traditional evaluations which is a known problem, but it does not clearly demonstrate that MIRA performs better than existing automated evaluation methods.

The central baseline and examples do not currently establish valid measurement of policy violations, as the traces themselves are not necessarily the only correct path. Also, the paper repeatedly describes τ²-bench as reporting 100% policy adherence, but §3 and Appendix C acknowledge that this is the authors’ own re-derived quantity, not a published policy-adherence submetric. More seriously, Appendix M’s supposedly unambiguous examples include policy interpretations that conflict with the official retail policy currently available: pending orders may have their payment method changed, returns may be refunded to an existing gift card, and item modifications are explicitly collected into one call. This does not prove those particular trajectories are compliant, because other conditions or a different policy revision may apply; however, the stated reasons are insufficient and undermine the examples offered to validate the judge. The exact benchmark revision, scoring derivation, applicable clauses, and independently adjudicated traces are essential before the headline gap can be trusted.

Several central conclusions overstate what the experimental design identifies. All three MIRA dimensions are judge-derived under stubbed tools, yet the 27.1% conditional result is described as passing end-state and action checks (§6.1; Appendix D). The τ²-to-MIRA comparison also changes the agent cohort and the grading mode from holistic to checklist, so it does not isolate mining. Finally, Figure 2 labels its first statistic “single trial” although the caption defines it as success at least once in three trials, and concrete user requests are regenerated across trials. These results cannot establish the claimed single-trial overestimation or agent-only inconsistency without separating sampling definitions, user variation, and attainability under the six-turn cap.

Questions for authors

For the reported 27.1% failure rate, were any actual environment-based checks used to verify the final state and actions? If all three evaluation results come from LLM judges, I think the paper should make this clearer. It would also be helpful to independently validate these cases, especially those where human annotators disagree.

Could the authors report the actual success rate for a single trial, as well as the rate of succeeding in all three trials? I would also like to see whether allowing more conversation turns changes the results, preferably using the same user goals across trials. Finally, for the mining ablation, could the authors keep the policy information and grading method consistent, and directly evaluate policy adherence rather than only task completion?

1 comment
AuthorsOpenReview ↗

you were right about Appendix M, and it led us to re-check all 127 failing cases against the policy text clause by clause — none of them fails on a reason the policy contradicts, so the 72.1% result stands unchanged.

One correction first, because you found a real error and we would rather state it plainly.

You are right about Appendix M: a pending order's payment method can be changed, a return refund can go to an existing gift card, and putting all item changes in one call is required rather than forbidden. Looking into why, we found something worse that we will also report — three of those four examples came from an earlier judging run by different judges, not the judge whose scores the paper reports. Under the reported judge all three pass, scoring 0.85 to 1.00.

So rather than swap in better examples, we re-checked everything: we pinned the policy text and went through all 127 conversations the reported judge marked as failing policy, clause by clause. None fails on a reason the policy contradicts, and your three mistaken readings appear zero times. The judge in fact draws a finer line than the objection — it limits refunds to the original payment method only for cancellations, where the policy does say that, and allows a gift card for returns, where the policy allows it. The headline was always computed from the reported judge and is unchanged: 72.1% is 329 of 456. The revision will replace the examples with ones from those 127, cite each to a policy line, and state the correction openly.

Q1. For the 27.1% figure, were any actual environment-based checks used to verify the final state and actions?

If all three come from LLM judges, the paper should say so. It would also help to independently validate these cases, especially where annotators disagree. No. Tool responses are fixed synthetic values, not a live system, so there is no final database to assert against and no reference action trace to compare to — all three scores come from the judge reading the conversation, and the revision will say so directly. The result should be read as one judge disagreeing with itself across three dimensions on the same conversation. We think that is still worth reporting, because the three things come apart under one grader, but it does not show a deterministic benchmark would have passed those runs and we will stop implying it. The figure also becomes 26.8% (60 of 224) once the failed judge call is removed — the same thing behind the 149 you flagged. On scope we should be straightforward: we compare against τ²-bench and against a change of grader on fixed conversations, which shows an end-state check leaves policy adherence unmeasured. It does not show MIRA beats other automated evaluators, and the revision will not suggest it does.

Q2. Could you report the actual single-trial success rate and the all-three-trials rate, say whether more turns changes the results using the same user goals, and for the ablation keep policy and grading consistent while directly evaluating policy adherence?

Scoring each trial on its own, the rate at which a model passes all three dimensions is 8.0%, 6.9% and 6.9% (60, 52 and 52 of 750), against pass¹ up to 24.0% and pass³ never above 2.7%. You were also right that the panel labels contradicted the caption; they will read "≥1 of 3 / ≥2 of 3 / all 3 trials", with the per-trial rates alongside. Only one of the two gaps is a reliability claim: going from "at least once" to "every time" costs most of the measured performance, while a single trial sitting below "at least once" is arithmetic. On turn budget we have not run it and will not guess. Every conversation runs to the six-turn limit, so a false promise in a final turn could be truncation; separating that from agent and user-simulator variation needs a re-run with the user turns held fixed, which we will list as a limitation.

On the ablation, the hand-authored scenarios are already re-scored with the current checklist for the three shared models, leaving one judge-matched pass of about 30 minutes. We will post it in this period. We will say in advance that the ablation we already have, on task completion, works against our own mining claim — hand-authored templates separate models more widely, 45.3 against 22.7 points. You were also right that the comparison changes cohort and grading mode together, and we will label it a combined effect. One correction to the direction, our fault for never writing it down: on MIRA the judge marks each mined rule pass or fail and the rubric folds those into three values, while τ² has no per-task rules so the judge gives one score taking 21 values. The checklist side is the coarser one at the threshold.

Given that the headline result survives a clause-level re-check of all 127 failing cases, that the provider-disjoint judge makes the finding stronger rather than weaker, and that the specific errors you identified will be corrected openly in the revision, we hope you would consider revisiting the soundness score.

Reviewer_dTbsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MIRA, a methodology for evaluating tool-augmented conversational agents on policy adherence by mining workflow graphs and ordering constraints from agent traces. The core claim is that existing end-state–based benchmarks (e.g., τ²-bench) use task success rate as an unsound proxy for policy adherence and that real policy adherence when evaluated directly is significantly lower.

Overall the paper makes an interesting and timely contribution. This core finding is relevant for both research in more robust evals and also has significance for industry practitioners. However, I have some doubts about the scope of MIRA and the novelty - see questions and weaknesses.

Critical strengths

S1: The central claim of the paper, that end-state correctness is not a sound proxy for policy adherence is sound and valid.

S2: The proposed bottom-up workflow and policy constraints mining from action-annotated transcripts (rather than relying on top-down authored policies) is novel (to my best knowledge) and interesting.

S3: The experimental results show significant gaps between existing evals and the MIRA evals which point to weaknesses in current evals and agents

S4: The paper is well written and the central claims are clearly articulated and the methods and experiments are mostly easy to follow

Critical weaknesses

W1: I would be interested in a more detailed discussion what types of policies the MIRA method can mine from existing traces and where its limitations are. MIRA starts with mining a canonical tool sequence and assumes that this is the correct path - this seems to be an assumption. If the majority of traces in the dataset are wrong, then MIRA would create a wrong canonical tool sequence, correct?

W2: l. 203 "rule-based detectors tag anomaly patterns" -> where do these detectors come from? it seems that you still require some domain-expert authored knowledge in this step?

W3: The policy rules seem to be mainly concerned with tool sequence - are there policy rules that you cannot learn with MIRA? I would assume yes, but a short discussion on this would be helpful. Can you for example learn a travel expense policy "you cannot expense costs for your family members" - if no such data of expenses of family members exists in the datasets, MIRA cannot mine such a rule (and it is also not a tool sequence constraint). in line 217 the authors say that the pipeline uses a policy knowledge base for non-ordering rules - so rules, so am I correct that rules like "you cannot expense costs for your family members" are out of scope for MIRA?

W4: I have a question /concerns regarding the novelty. The idea if "grading notes" that capture policy checks is not new in agent evaluation (eg Chong et al 2026). The main contribution of MIRA, to my understanding, is therefore in the fact that the policy is mined from data (and therefore better/more comprehensive, etc) and not so much the fact that no other work is checking the agent against a policy with LLM-as-a-judge. Can the authors confirm/clarify this point in the rebuttal?

Cong et al. 2026. TALK, EVALUATE, DIAGNOSE: USER-AWARE AGENT EVALUATION WITH AUTOMATED ERROR ANALYSIS. ICLR

Questions for authors

Q1: l 261: the scores are {0, 0.5, 1} and the threshold is 0.7 - is the threshold applied at an aggregate (eg the average score over the test set) or at each test instance? If it is the later, the {0, 0.5, 1} score would collapse to binary outcomes

Q2: is table 1 an average over all models and is figure 3 the corresponding per-model view? As the scores between models differ a lot, the average numbers in table 1 are hiding a lot of details

Q3: what is the metric of the pass^k experiments in figure 2? policy adherence, task success or both?

1 comment
AuthorsOpenReview ↗

mining from data is one of our two claims; the other is measuring policy adherence in a way that can be checked, which is what TED does not set out to do.

On novelty (W4). Thank you for the TED comparison — it sharpened the positioning. Mining is one of our two claims; the other is measuring policy adherence in a way that can be checked. We do not claim the judging mechanism. TED already has an LLM judge reading a trajectory against a plain-English checklist, that checklist already covers tool calls and their order, its judge works from the log alone, and it has personas too. The revision will say so.

What TED does not do:

  • treat policy adherence as its own score — it checks whether subgoals were reached, so nothing checks confirming before acting, verifying identity, or promising what the agent never did.
  • compare its grading against an end-state grader, which is the comparison that gives us 100% → 72.1% on fixed conversations;
  • mine scenarios from real annotated dialogues — TED adapts existing datasets by hand;
  • validate its judge against people — it checks whether repeated votes agree, where we use 190 conversations labelled by two humans each;
  • report pass³ — TED reports best-of-k, so a reliability collapse stays invisible.

W1–W3: where mining stops. We will state the limits rather than defend them. The canonical sequence is only the most common path; on a corpus where most behaviour breaks the rules it would encode the wrong path, and no amount of judge quality catches that. The detectors are defined against the mined graph, so they carry to a new corpus, but the pipeline still needs a tool list and a policy knowledge base — the paper already says bottom-up construction relocates domain artifacts rather than removing them. Your expense example is out of scope twice over: it is not about ordering, and if nobody in the corpus tried it there is no signal to learn from. Mining recovers procedure, not policy in general.

Q1. Is the 0.7 threshold applied per instance or to an aggregate?

Per conversation, so your conclusion follows — for pass/fail the three-point scale is effectively binary. Across the 2,250 policy scores, 650 are 1.0, 1,498 are 0.0, and the 102 that are 0.5 fail along with the zeros. The half score still does work elsewhere, which is why we keep it: the human-agreement statistics use the full scale, so a 1.0-versus-0.5 disagreement counts as half wrong rather than fully wrong. The revision will state both. Your question also surfaced something we had not written down and will add: the two rows of Table 1 use the same threshold but very different granularity — three values on the MIRA side, 21 on the τ² side.

Q2. Is Table 1 an average over all models, with Figure 3 the per-model view?

Yes to both, and you are right that the average hides a lot. Policy adherence runs from 12.2% to 40.9% across the five models and the overall pass rate from 2.7% to 10.9%. The revision will put the range in the caption with a pointer to the per-model figure. Worth adding that the spread is itself a result rather than noise: an end-state proxy reports no variation at all on these models. The size depends on the judge — 28.7 points with ours, 18.0 with the second, 13.8 with a third — though the model ordering does not, and we report all three rather than the most flattering.

Q3. What is the metric in the pass^k experiments?

All three dimensions together. A scenario counts as passed in a trial only if task completion, policy adherence and tool correctness all pass in that same conversation — a conjunction, not an average — which is why pass^k sits well below any single dimension. pass^k is the share of a model's scenarios passed on all three in at least k of three trials. The caption says this but the figure's panel labels contradicted it, as another reviewer noticed; we will fix the labels and add the genuine per-trial rates.

Reviewer_TLmTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether reaching the correct final state is enough to establish that a tool-using agent followed policy throughout an interaction. To examine this, the authors regrade released tau2-bench transcripts and find policy violations that the end-state proxy misses. They then introduce MIRA, which mines workflows from customer-service dialogues, builds evaluation scenarios, and grades full transcripts for task completion, policy adherence, and tool correctness. The experiments show that agents can complete tasks while violating policy, and that reliable success across repeated trials remains rare.

Critical strengths
  • Regrading the same released tau2-bench transcripts holds agent behavior fixed while changing the evaluation. This provides a direct example of policy violations that an end-state proxy can miss.
  • The judge analysis includes agreement on model ordering across providers and measures same-provider leniency. The repeated-trial evaluation also reports uncertainty and avoids claiming a reliable model ordering where the evidence is weak.
Critical weaknesses
  • The headline comparison mixes results from different settings. The paper contrasts average policy adherence on MIRA with perfect adherence under an end-state proxy, yet it does not apply an end-state grader to the MIRA scenarios. The tau2 proxy score is also a re-derivation by the authors rather than a published benchmark sub-metric. The same-transcript tau2 analysis is useful, but the broader comparison needs clearer qualification.

  • The benefit of mining workflows is not well isolated. The mining ablation measures task completion with limited trials and gives mixed results across agents (Table 11). It does not establish that mining improves the detection of policy violations, while Appendix L attributes the separation to the grading dimension. Stronger evidence is needed to distinguish the contribution of mined scenarios from that of transcript-level grading.

  • The readability is somehow very limited due to excessive numbers in the presentation.

Questions for authors
  1. How is the perfect policy-adherence proxy score derived from the released tau2 artifacts?
  2. What share of fabricated completions involves escalation claims, and do the main conclusions hold when these are reported separately? Please distinguish promises to escalate from claims that escalation has already occurred.
  3. Among runs that pass task completion and tool correctness but fail policy adherence, how many involve fabrication and how many receive only a partial policy score?
1 comment
AuthorsOpenReview ↗

the headline rests on one single-factor experiment rather than the mixed comparison and we will post the policy-adherence ablation you asked for in this period

Thank you — all three questions are answered below with numbers from the released data, and two of your weaknesses led to corrections we will make.

Q1. How is the 100% policy-adherence proxy score derived from the released τ² artifacts?

It is ours, not a published τ² metric, and the revision will say so where the number appears. All 1,824 released τ²-retail conversations pass the benchmark's own communicate_checks, and those checks are the only part of the τ² reward that looks at what the agent said and did rather than at the final database state — so the rate it implies is 100%. If you would rather say τ² does not measure policy adherence at all, that is our point too. You are also right that the broad MIRA-vs-τ² comparison mixes settings, and we will stop leaning on it. The claim rests on one clean experiment: take τ²-bench's own conversations and change only the grading — same conversations, same agents, same judge model — and policy adherence falls from 100% to 72.1%. The wider comparison changes scenarios, agents and judge together, and we will label it the combined effect of the mined setup. The fourth table cell we cannot fill: end-state checks are written by hand per task, and none exist for scenarios τ² never defined.

Q2. What share of fabricated completions involves escalation claims, do the conclusions hold if reported separately, and can you separate promises to escalate from claims it already happened?

644 of the 1,823 fabricated-completion violations mention escalation — 35.3% of that class, 20.6% of all 3,129 hard violations. Reporting them separately changes nothing: drop every escalation case and fabricated completion is still 1,179 violations, 37.7% of all hard violations, more than twice the next category (identity checks skipped, 17.1%). Promises versus completed claims we cannot tell apart from what we stored. The category is defined as promising an action and never calling the tool, so it covers both wordings and the rubric never recorded which. Matching phrases afterwards sorts only 122 of the 644, so we will mention it as a limitation rather than split the number on a weak signal. We should also flag that our classifier takes the first matching keyword and checks "claim" before "escalation", so the fabrication bucket absorbs escalation cases a different order would separate. On isolating mining, you were right and we were wrong. Two numbers in §7 were printed the wrong way round: hand-authored templates give the wider spread on task completion, 45.3 points against 22.7, as the appendix table's own columns show. We will correct them and the conclusion drawn from them. What we keep is the appendix's own claim — mining is what makes the measurement possible, since it produces the per-scenario ordering rules there is otherwise nothing to check against, while the size of the gap comes from the grading axis. Your underlying ask we can run: the hand-authored scenarios are already re-scored with the current checklist for the three shared models, leaving one judge-matched pass of about 30 minutes. We will post it in this period either way.

Q3. Among runs that pass task completion and tool correctness but fail policy adherence, how many involve fabrication and how many receive only a partial score?

Of those 60: 52 (86.7%) score zero on policy, 8 (13.3%) get the half score, 24 (40.0%) mention fabrication. So the result does not rest on partial credit. We also found an error we will correct: one run's judge call failed and was saved as a zero, which we counted as a violation. Removing it gives 26.8% (60 of 224) rather than 27.1% (61 of 225), and the same failed call is why one table showed 149 — one harness failure behind both accounting problems. A second judge gives 28.8% on the same runs.

Given the corrected ablation numbers, the corrected conditional, and the ablation result we will post in this period, we hope you would consider revisiting your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.