- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses unreliable evaluation of task-oriented dialogue systems caused by user simulators that deviate from their assigned goals or profiles. The authors introduce TestTOD, a framework that classifies dialogue executions as Pass, Fail, or Inconclusive and counts only faithful executions toward validated test coverage. Building on this framework, CoverSim uses GEPA to optimize simulator prompts based on feedback about profile violations, improving simulation faithfulness without updating model weights. The authors evaluate the approach on the math tutoring and document creation domains of SimulatorArena, comparing it with original prompts, self-refinement, secondary-corrector methods, and fine-tuned simulators. They validate its effectiveness through evaluations involving multiple models and controlled profile combinations. In math tutoring, profile coherence improves by up to 9.5 percentage points, and full-profile coverage more than doubles for all three simulators compared with their original versions, revealing previously hidden agent failures. This paper’s main contribution is to make validated test coverage an objective for user simulator design and evaluation, showing that overall task success alone is insufficient to reliably assess dialogue systems.
Critical strengths
- Research Background: Thsi paper addresses a practical problem: user simulators that fail to follow their assigned goals or profiles can distort evaluation results. This provides a clear research motivation.
- Method: CoverSim establishes a closed loop of detecting profile violations, providing feedback, and optimizing simulator prompts, without updating model weights.
- Experiments: The experiments cover a broad range of baselines, including original prompts, self-refinement, secondary-corrector methods, and training-based approaches. They also report profile coherence, test coverage, and accuracy on covered executions, providing a more informative evaluation than overall task success alone.
Critical weaknesses
- Limited algorithmic novelty. Lines 238–265 show that CoverSim mainly applies the existing GEPA method to optimize simulator prompts using feedback on profile violations. Its contribution lies primarily in the coverage perspective and evaluation framework. Ablations are needed to establish the additional benefit of its specific feedback design over standard profile-consistency optimization.
- Lack of independent validation. Lines 162–165 and 257–265 indicate that the same oracles guide optimization and evaluate its outcomes. Table 3 does not report agreement with human judgments on newly generated dialogues after optimization. Thus, the gains may partly reflect adaptation to the oracles’ preferences. Human blind evaluation or validation with independent oracles would strengthen the evidence.
- The coverage experiment lacks an equal-budget control. Lines 419–425 and Figure 3 compare the original simulator’s single-round coverage of 44% with cumulative coverage of 88% after additional retries using the optimized simulator. Without giving the original simulator the same retries, the experiment cannot separate the benefits of prompt optimization from those of additional attempts.
If these concerns are adequately addressed, I would consider raising my score.
Questions for authors
I have no specific questions.
4 comments
Response 2
C2 - Does the simulator overfit the oracle used for optimization?
We thank the reviewer for raising this question. After performing additional evaluations we show that the coverage gains are not specific to the optimization oracle.
To study this, we added two new evaluations that were not used during simulator optimization. The first measures the same testing metrics with a different LLM judge, to check whether the gains depend on the original judge. The second measures human-likeness, a different property altogether, to check whether improvements extend beyond the optimization metric. The independent judge confirms coverage improvements across all three simulators; for example, DeepSeek’s full-profile coverage stays high under both judges (18% / 34%). CoverSim also improves human-likeness across all three simulators: for DeepSeek, the judge’s accuracy in distinguishing simulated from real conversations drops from 65.5% to 48.5% (50% as the best). Notably, improved human-likeness emerges without directly optimizing for it. Together, these results provide evidence against overfitting to either the original judge or the optimization metric: the gains extend to a different judge and a different evaluation criterion.
Experiment details.
For coverage, we evaluated the same 100 conversations per simulator with the original DeepSeek analyzer and an independent Grok analyzer. Under the DeepSeek / Grok analyzers, full-profile coverage increased from 9% / 18% to 18% / 34% for DeepSeek, 10% / 20% to 20% / 27% for Grok, and 4% / 3% to 12% / 17% for Claude. Our coverage improvements remain valid on the independent Grok analyzer, which is not used during the GEPA optimization, proving we are not overfitting to the judge. The coverage numbers from the Grok analyzer are higher because it has a small bias toward longer conversations.
Independent-analyzer results (DeepSeek / Grok; %; 100 conversations per simulator)
| Model | Full coherence | IS coherence | WS coherence | Full coverage | IS coverage | WS coverage | Turns |
|---|---|---|---|---|---|---|---|
| DeepSeek original | 82.1 / 83.3 | 76.4 / 78.1 | 89.8 / 90.2 | 9 / 18 | 13 / 36 | 36 / 46 | 6.3 |
| Grok original | 85.6 / 86.2 | 83.4 / 83.5 | 88.6 / 89.8 | 10 / 20 | 26 / 43 | 31 / 46 | 6.8 |
| Claude original | 72.5 / 73.7 | 66.2 / 65.7 | 80.8 / 84.3 | 4 / 3 | 8 / 19 | 11 / 21 | 6.2 |
| DeepSeek self-refinement | 75.8 / 75.4 | 68.9 / 65.6 | 84.9 / 88.5 | 6 / 15 | 13 / 23 | 23 / 43 | 4.0 |
| Grok self-refinement | 85.7 / 82.3 | 81.9 / 75.9 | 90.7 / 90.8 | 17 / 16 | 23 / 26 | 42 / 46 | 3.6 |
| Claude self-refinement | 73.1 / 75.7 | 68.5 / 68.8 | 79.3 / 84.8 | 6 / 9 | 9 / 24 | 16 / 30 | 3.8 |
| DeepSeek secondary corrector | 83.1 / 86.9 | 78.2 / 83.6 | 89.6 / 91.3 | 7 / 23 | 22 / 45 | 33 / 46 | 6.6 |
| Grok secondary corrector | 85.9 / 85.1 | 83.9 / 81.3 | 88.8 / 90.2 | 12 / 19 | 29 / 38 | 29 / 42 | 6.8 |
| Claude secondary corrector | 76.5 / 77.7 | 69.3 / 70.3 | 86.0 / 87.5 | 6 / 11 | 16 / 24 | 22 / 33 | 6.4 |
| USP SFT | 81.8 / 88.5 | 80.0 / 90.2 | 84.2 / 86.2 | 6 / 20 | 24 / 62 | 24 / 34 | 9.9 |
| USP SFT+RLCC | 83.6 / 89.1 | 81.4 / 90.7 | 86.5 / 87.1 | 4 / 28 | 22 / 60 | 21 / 40 | 9.9 |
| DeepSeek + CoverSim | 89.0 / 89.8 | 87.5 / 86.9 | 91.1 / 93.6 | 18 / 34 | 38 / 51 | 37 / 61 | 7.5 |
| Grok + CoverSim | 90.7 / 90.8 | 89.9 / 90.4 | 91.8 / 91.4 | 20 / 27 | 37 / 56 | 41 / 49 | 7.4 |
| Claude + CoverSim | 83.2 / 83.1 | 80.9 / 80.3 | 86.3 / 86.8 | 12 / 17 | 30 / 36 | 23 / 35 | 6.4 |
For human-likeness, we use SimulatorArena’s conversation-level Turing test. A DeepSeek judge identifies the real conversation from a real–simulated pair on the same problem; we evaluate 100 pairs per simulator in both orders. Judge accuracy changes from 65.5% to 48.5% for DeepSeek, 75.0% to 64.0% for Grok, and 59.5% to 53.0% for Claude (closer to 50% means better).
Human-likeness Turing test (100 pairs; both orders)
| Simulator | Original judge accuracy p (%) | Original |p − 50| | Original turns | CoverSim judge accuracy p (%) | CoverSim |p − 50| | CoverSim turns |
|---|---|---|---|---|---|---|
| DeepSeek Flash | 65.5 | 15.5 | 5.6 | 48.5 | 1.5 | 6.8 |
| Grok 4.3 | 75.0 | 25.0 | 5.9 | 64.0 | 14.0 | 6.0 |
| Claude Haiku 4.5 | 59.5 | 9.5 | 5.7 | 53.0 | 3.0 | 5.2 |
The full results are in the tables above. The second judge provides an independent check, though the judges may share biases; the human-likeness test is an automated proxy rather than a human study.
Response 3 - The End
C3 - Does CoverSim improve coverage under an equal budget?
We thank the reviewer for raising this concern. We evaluated both the original simulator and CoverSim under the same budget of three attempts per profile. CoverSim consistently covers more of the grid: across three attempts, it produces a conclusive test for 23/25 profiles, compared with 15/25 for the original simulator. Thus, the coverage gain persists under an equal budget.
Experiment details.
We evaluated both simulators on a new 5×5 grid crossing Answer Thoroughness and Feedback Style. Each cell represents a combination of the two profile features. A test is conclusive when the simulated user follows both assigned features and the outcome can be attributed to the tutor. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. In the three individual runs, the original simulator covers 12, 10, and 12 cells, while CoverSim covers 19, 18, and 20. Across all three attempts, 10 cells remain inconclusive for the original simulator versus 2 for CoverSim. The per-run heatmaps and per-cell counts are in Figure 3 and Section 5.4 of the updated paper.
Grid results: conclusive cells out of 25
| Simulator prompt | Run 1 | Run 2 | Run 3 | Mean ± SD | At least once | All three | None |
|---|---|---|---|---|---|---|---|
| Original | 12 | 10 | 12 | 11.3 ± 1.2 | 15 | 6 | 10 |
| CoverSim | 19 | 18 | 20 | 19.0 ± 1.0 | 23 | 14 | 2 |
This is a different grid from the one in the paper, so it does not directly test retries on the original 14 inconclusive cells. Like the original grid, it is a proof of concept for inspecting coverage across combinations of user characteristics.
Thank you for the additional experiments. The equal-budget comparison addresses my concern about retries. With three attempts per profile, CoverSim produced conclusive tests for 23 of 25 profiles on the new grid, compared with 15 for the original simulator. The independent judge also found coverage gains for all three simulators, which reduces my concern that the improvements depend on the judge used during optimization.
I see value in TestTOD’s formulation of validated coverage as a testing objective. My remaining concern is how much CoverSim’s specific feedback design contributes to the gains. An ablation comparing feedback on specific profile violations with a general profile-consistency signal, using the same GEPA setup, would help isolate that contribution. On balance, I maintain my rating of 2.
Author's Response 1
We thank the reviewer for recognizing the importance of reliable simulator-based evaluation, the closed-loop design of CoverSim, and the value of reporting coverage alongside agent accuracy. We address reviewer’s concerns about the technical contribution, independent validation, and equal-budget comparisons below.
C1 - CoverSim applies an existing prompt optimizer, limiting its algorithmic novelty.
We agree that GEPA itself is not a contribution of this work. In fact, we do not claim a technical contribution in prompt optimization. Our main contribution is a methodology for testing and user simulation for task-oriented dialogue systems.
Prior simulator evaluations often emphasize human-likeness or similarity to real conversations, which does not tell us whether a TOD system has been tested under specified user profiles. Our paper is the first to formulate evaluation as testing across user characteristics, enabling us to measure coverage and design simulators with a testing objective in mind. We believe that defining the right evaluation and objectives can be more valuable than a new optimization technique: it gives the community something meaningful to optimize.
CoverSim is a simple instantiation of this methodology. Its technical simplicity is a feature, not a bug! Using prompt optimization such as GEPA, we can use off-the-shelf closed-source LLMs to test TOD systems and demonstrate improved testing coverage over prior approaches. The same methodology could be applied with other simulator optimization methods, such as SFT or RL for open-weight models. We leave that choice to the user; our goal is to show that even a simple instantiation already delivers better testing.