acceptodds
Under review as a conference paper at ICLR 2027

CoverSim: Coverage-Guided User Simulation for Task-Oriented Dialogue

Abstract

Task-oriented dialogue systems are increasingly evaluated with LLM-simulated users, enabling scalable testing across diverse user goals and profiles. Yet current approaches often reward human-likeness without verifying whether the simulator actually follows the user condition it is meant to enact. As a result, a conversation can look realistic while failing to test the intended behavior of the system. In this work, we ask what makes a user simulator good for testing task-oriented dialogue systems. We formalize simulation-based evaluation as testing, where each goal–profile pair is a test input, but it counts toward coverage only if the simulator faithfully executes it. This turns simulator improvement into a coverage-optimization problem: reduce inconclusive executions so more goal–profile conditions become testable. We then optimize user simulators directly for this objective using evolutionary prompt optimization, producing simulators that are better at testing the system, not merely imitating the user. We instantiate our automated testing framework using the data and user profiles from SimulatorArena and show that our simulator optimization strategy improves profile-following accuracy by up to 9.5% and more than doubles validated test coverage. We then show that standard TOD evaluation hides both agent failures and simulator failures, yielding a misleading picture of agent performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
task-oriented dialogue, dialogue systems, user simulation, dialogue evaluation, llm-based evaluation, human-ai interaction, prompt optimization, conversational agents, interactive systems, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QiHZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses unreliable evaluation of task-oriented dialogue systems caused by user simulators that deviate from their assigned goals or profiles. The authors introduce TestTOD, a framework that classifies dialogue executions as Pass, Fail, or Inconclusive and counts only faithful executions toward validated test coverage. Building on this framework, CoverSim uses GEPA to optimize simulator prompts based on feedback about profile violations, improving simulation faithfulness without updating model weights. The authors evaluate the approach on the math tutoring and document creation domains of SimulatorArena, comparing it with original prompts, self-refinement, secondary-corrector methods, and fine-tuned simulators. They validate its effectiveness through evaluations involving multiple models and controlled profile combinations. In math tutoring, profile coherence improves by up to 9.5 percentage points, and full-profile coverage more than doubles for all three simulators compared with their original versions, revealing previously hidden agent failures. This paper’s main contribution is to make validated test coverage an objective for user simulator design and evaluation, showing that overall task success alone is insufficient to reliably assess dialogue systems.

Critical strengths
  1. Research Background: Thsi paper addresses a practical problem: user simulators that fail to follow their assigned goals or profiles can distort evaluation results. This provides a clear research motivation.
  2. Method: CoverSim establishes a closed loop of detecting profile violations, providing feedback, and optimizing simulator prompts, without updating model weights.
  3. Experiments: The experiments cover a broad range of baselines, including original prompts, self-refinement, secondary-corrector methods, and training-based approaches. They also report profile coherence, test coverage, and accuracy on covered executions, providing a more informative evaluation than overall task success alone.
Critical weaknesses
  1. Limited algorithmic novelty. Lines 238–265 show that CoverSim mainly applies the existing GEPA method to optimize simulator prompts using feedback on profile violations. Its contribution lies primarily in the coverage perspective and evaluation framework. Ablations are needed to establish the additional benefit of its specific feedback design over standard profile-consistency optimization.
  2. Lack of independent validation. Lines 162–165 and 257–265 indicate that the same oracles guide optimization and evaluate its outcomes. Table 3 does not report agreement with human judgments on newly generated dialogues after optimization. Thus, the gains may partly reflect adaptation to the oracles’ preferences. Human blind evaluation or validation with independent oracles would strengthen the evidence.
  3. The coverage experiment lacks an equal-budget control. Lines 419–425 and Figure 3 compare the original simulator’s single-round coverage of 44% with cumulative coverage of 88% after additional retries using the optimized simulator. Without giving the original simulator the same retries, the experiment cannot separate the benefits of prompt optimization from those of additional attempts.

If these concerns are adequately addressed, I would consider raising my score.

Questions for authors

I have no specific questions.

4 comments
AuthorsOpenReview ↗

Author's Response 1

We thank the reviewer for recognizing the importance of reliable simulator-based evaluation, the closed-loop design of CoverSim, and the value of reporting coverage alongside agent accuracy. We address reviewer’s concerns about the technical contribution, independent validation, and equal-budget comparisons below.

C1 - CoverSim applies an existing prompt optimizer, limiting its algorithmic novelty.

We agree that GEPA itself is not a contribution of this work. In fact, we do not claim a technical contribution in prompt optimization. Our main contribution is a methodology for testing and user simulation for task-oriented dialogue systems.

Prior simulator evaluations often emphasize human-likeness or similarity to real conversations, which does not tell us whether a TOD system has been tested under specified user profiles. Our paper is the first to formulate evaluation as testing across user characteristics, enabling us to measure coverage and design simulators with a testing objective in mind. We believe that defining the right evaluation and objectives can be more valuable than a new optimization technique: it gives the community something meaningful to optimize.

CoverSim is a simple instantiation of this methodology. Its technical simplicity is a feature, not a bug! Using prompt optimization such as GEPA, we can use off-the-shelf closed-source LLMs to test TOD systems and demonstrate improved testing coverage over prior approaches. The same methodology could be applied with other simulator optimization methods, such as SFT or RL for open-weight models. We leave that choice to the user; our goal is to show that even a simple instantiation already delivers better testing.

AuthorsOpenReview ↗

Response 2

C2 - Does the simulator overfit the oracle used for optimization?

We thank the reviewer for raising this question. After performing additional evaluations we show that the coverage gains are not specific to the optimization oracle.

To study this, we added two new evaluations that were not used during simulator optimization. The first measures the same testing metrics with a different LLM judge, to check whether the gains depend on the original judge. The second measures human-likeness, a different property altogether, to check whether improvements extend beyond the optimization metric. The independent judge confirms coverage improvements across all three simulators; for example, DeepSeek’s full-profile coverage stays high under both judges (18% / 34%). CoverSim also improves human-likeness across all three simulators: for DeepSeek, the judge’s accuracy in distinguishing simulated from real conversations drops from 65.5% to 48.5% (50% as the best). Notably, improved human-likeness emerges without directly optimizing for it. Together, these results provide evidence against overfitting to either the original judge or the optimization metric: the gains extend to a different judge and a different evaluation criterion.

Experiment details.

For coverage, we evaluated the same 100 conversations per simulator with the original DeepSeek analyzer and an independent Grok analyzer. Under the DeepSeek / Grok analyzers, full-profile coverage increased from 9% / 18% to 18% / 34% for DeepSeek, 10% / 20% to 20% / 27% for Grok, and 4% / 3% to 12% / 17% for Claude. Our coverage improvements remain valid on the independent Grok analyzer, which is not used during the GEPA optimization, proving we are not overfitting to the judge. The coverage numbers from the Grok analyzer are higher because it has a small bias toward longer conversations.

Independent-analyzer results (DeepSeek / Grok; %; 100 conversations per simulator)

ModelFull coherenceIS coherenceWS coherenceFull coverageIS coverageWS coverageTurns
DeepSeek original82.1 / 83.376.4 / 78.189.8 / 90.29 / 1813 / 3636 / 466.3
Grok original85.6 / 86.283.4 / 83.588.6 / 89.810 / 2026 / 4331 / 466.8
Claude original72.5 / 73.766.2 / 65.780.8 / 84.34 / 38 / 1911 / 216.2
DeepSeek self-refinement75.8 / 75.468.9 / 65.684.9 / 88.56 / 1513 / 2323 / 434.0
Grok self-refinement85.7 / 82.381.9 / 75.990.7 / 90.817 / 1623 / 2642 / 463.6
Claude self-refinement73.1 / 75.768.5 / 68.879.3 / 84.86 / 99 / 2416 / 303.8
DeepSeek secondary corrector83.1 / 86.978.2 / 83.689.6 / 91.37 / 2322 / 4533 / 466.6
Grok secondary corrector85.9 / 85.183.9 / 81.388.8 / 90.212 / 1929 / 3829 / 426.8
Claude secondary corrector76.5 / 77.769.3 / 70.386.0 / 87.56 / 1116 / 2422 / 336.4
USP SFT81.8 / 88.580.0 / 90.284.2 / 86.26 / 2024 / 6224 / 349.9
USP SFT+RLCC83.6 / 89.181.4 / 90.786.5 / 87.14 / 2822 / 6021 / 409.9
DeepSeek + CoverSim89.0 / 89.887.5 / 86.991.1 / 93.618 / 3438 / 5137 / 617.5
Grok + CoverSim90.7 / 90.889.9 / 90.491.8 / 91.420 / 2737 / 5641 / 497.4
Claude + CoverSim83.2 / 83.180.9 / 80.386.3 / 86.812 / 1730 / 3623 / 356.4

For human-likeness, we use SimulatorArena’s conversation-level Turing test. A DeepSeek judge identifies the real conversation from a real–simulated pair on the same problem; we evaluate 100 pairs per simulator in both orders. Judge accuracy changes from 65.5% to 48.5% for DeepSeek, 75.0% to 64.0% for Grok, and 59.5% to 53.0% for Claude (closer to 50% means better).

Human-likeness Turing test (100 pairs; both orders)

SimulatorOriginal judge accuracy p (%)Original |p − 50|Original turnsCoverSim judge accuracy p (%)CoverSim |p − 50|CoverSim turns
DeepSeek Flash65.515.55.648.51.56.8
Grok 4.375.025.05.964.014.06.0
Claude Haiku 4.559.59.55.753.03.05.2

The full results are in the tables above. The second judge provides an independent check, though the judges may share biases; the human-likeness test is an automated proxy rather than a human study.

AuthorsOpenReview ↗

Response 3 - The End

C3 - Does CoverSim improve coverage under an equal budget?

We thank the reviewer for raising this concern. We evaluated both the original simulator and CoverSim under the same budget of three attempts per profile. CoverSim consistently covers more of the grid: across three attempts, it produces a conclusive test for 23/25 profiles, compared with 15/25 for the original simulator. Thus, the coverage gain persists under an equal budget.

Experiment details.

We evaluated both simulators on a new 5×5 grid crossing Answer Thoroughness and Feedback Style. Each cell represents a combination of the two profile features. A test is conclusive when the simulated user follows both assigned features and the outcome can be attributed to the tutor. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. In the three individual runs, the original simulator covers 12, 10, and 12 cells, while CoverSim covers 19, 18, and 20. Across all three attempts, 10 cells remain inconclusive for the original simulator versus 2 for CoverSim. The per-run heatmaps and per-cell counts are in Figure 3 and Section 5.4 of the updated paper.

Grid results: conclusive cells out of 25

Simulator promptRun 1Run 2Run 3Mean ± SDAt least onceAll threeNone
Original12101211.3 ± 1.215610
CoverSim19182019.0 ± 1.023142

This is a different grid from the one in the paper, so it does not directly test retries on the original 14 inconclusive cells. Like the original grid, it is a proof of concept for inspecting coverage across combinations of user characteristics.

Thank you for the additional experiments. The equal-budget comparison addresses my concern about retries. With three attempts per profile, CoverSim produced conclusive tests for 23 of 25 profiles on the new grid, compared with 15 for the original simulator. The independent judge also found coverage gains for all three simulators, which reduces my concern that the improvements depend on the judge used during optimization.

I see value in TestTOD’s formulation of validated coverage as a testing objective. My remaining concern is how much CoverSim’s specific feedback design contributes to the gains. An ablation comparing feedback on specific profile violations with a general profile-consistency signal, using the same GEPA setup, would help isolate that contribution. On balance, I maintain my rating of 2.

Reviewer_sL8hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work argues that when LLM-based user simulators are used to test task-oriented dialogue agents, a conversation constitutes a valid test only if the simulator faithfully enacts its assigned goal and profile; otherwise the execution is inconclusive. The authors introduce TestTOD, a framework with three LLM-based analyzers (goal following, profile consistency, task completion) whose prompts are optimized with GEPA on annotated SimulatorArena data. They then optimize the simulator prompt with GEPA to reduce profile-inconsistency verdicts (CoverSim). On math tutoring and document creation, they report up to a 9.5-point gain in profile coherence and roughly a doubling of fully coherent coverage (for example, 10% to 30%). They further claim that aggregate accuracy conceals agent differences and that a 5x5 profile grid improves from 11/25 to 22/25 validated cells.

Critical strengths
  1. The concept is simple and useful: each goal-profile pair is treated as a test input, and coverage is reported as the fraction of inputs actually exercised. The reported coverage of 6–12% for original simulators is a good demonstration that current simulator-based evaluations frequently fail to test the intended user conditions.
  2. The comparison is reasonable: It includes self-refinement, a secondary corrector, and a fine-tuned simulator as baselines, reports computational cost, and includes a second domain in which gains are honestly shown to be smaller.
Critical weaknesses

Several downstream are on small, uncontrolled samples. In Table 2 the covered sets contain roughly 25–31 conversations per agent and no confidence intervals are provided, however it states that gpt-5-mini clearly underperforms gpt-5 and that rankings change. Because coverage is conditioned on simulator behavior, covered cases may also be selected on difficulty. In the 5x5 grid experiment, only the 14 previously inconclusive cells were re-run, once each, with the optimized simulator and without a matched re-run of the original simulator, so part of the apparent gain (including the two newly exposed failures) may be attributable to execution variability. The relationship between the 50 optimization profiles and the test profiles is not clearly specified.

Questions for authors
  1. Do any of the 50 optimization profiles overlap with the 100 test pairs, and how many distinct profiles are in the test set?
  2. In the 5x5 grid, what happens if the 14 inconclusive cells are re-run several times with the original simulator?
  3. Please add confidence intervals to Tables 1 and 2, and show which agent differences remain.
2 comments
AuthorsOpenReview ↗

Author's Response 1

We thank the reviewer for recognizing the value of treating goal–profile pairs as test inputs, reporting validated coverage, and evaluating CoverSim against a broad set of baselines. We address the reviewer’s concerns and questions below.

C1 - Do the optimization and test profiles overlap?

No. The 50 profiles used to optimize the simulator are drawn from the training set and are distinct from the test profiles. The test set contains 100 distinct profiles, with no duplicates, and every agent is evaluated on the same test cases.

C2 - Are the coverage-grid gains due to unequal numbers of attempts?

We thank the reviewer for raising this concern. We evaluated both the original simulator and CoverSim under the same budget of three attempts per profile. CoverSim consistently covers more of the grid: across three attempts, it produces a conclusive test for 23/25 profiles, compared with 15/25 for the original simulator. Thus, the coverage gain persists under an equal budget.

Experiment details.

We evaluated both simulators on a new 5×5 grid spanning two user profile features - Answer Thoroughness and Feedback Style. Each cell represents a combination of the two profile features. A test is conclusive when the simulated user follows both assigned features and the outcome can be attributed to the tutor. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. In the three individual runs, the original simulator covers 12, 10, and 12 cells, while CoverSim covers 19, 18, and 20. Across all three attempts, 10 cells remain inconclusive for the original simulator versus 2 for CoverSim. The per-run heatmaps and per-cell counts are in Figure 3 and Section 5.4 of the updated paper.

Grid results: conclusive cells out of 25

Simulator promptRun 1Run 2Run 3Mean ± SDAt least onceAll threeNone
Original12101211.3 ± 1.215610
CoverSim19182019.0 ± 1.023142

This is a different grid from the one in the paper, so it does not directly test retries on the original 14 inconclusive cells. Like the original grid, it is a proof of concept for inspecting coverage across combinations of user characteristics.

AuthorsOpenReview ↗

Response 2

C3 — Do the results and agent differences hold across runs?

We thank the reviewer for this suggestion. We repeated the math-tutoring evaluation three times and added 95% confidence intervals. The results are consistent across runs: CoverSim improves full-profile coherence for all simulator models in every run. For DeepSeek, full-profile coverage increases from 7.7 ± 2.9% to 20.0 ± 5.0% across runs. The repeated evaluation also shows a substantial difference between GPT-5 and GPT-5-mini on covered tests: under the CoverSim-optimized simulator, their Acc@Covered is 91.7 ± 5.0% and 69.4 ± 12.0%, respectively. This supports the finding that GPT-5 outperforms GPT-5-mini on the covered tests across the repeated evaluation.

Experiment details.

Each run contains 100 conversations per setting, and every agent is evaluated on the same 100 distinct test profiles. Results are means ± half-width of the 95% confidence interval across three runs, calculated using Student’s t. Under CoverSim, the proportion of conclusive tests is 15.7 ± 7.6% for GPT-5 and 16.0 ± 8.6% for GPT-5-mini. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. The full results are in the tables below. The repeated experiments use the updated DeepSeek snapshot because the original snapshot is no longer served.

Simulator results: mean ± 95% CI (three runs)

ModelFull coherenceIS coherenceWS coherenceFull coverageIS coverageWS coverage
DeepSeek original81.7 ± 1.176.0 ± 1.089.2 ± 2.07.7 ± 2.914.7 ± 3.833.0 ± 9.0
Grok original86.1 ± 1.484.3 ± 2.088.4 ± 1.511.0 ± 11.427.3 ± 5.732.0 ± 13.8
Claude original72.5 ± 2.166.1 ± 2.581.2 ± 1.84.0 ± 2.511.0 ± 7.510.7 ± 3.8
DeepSeek self-refinement76.8 ± 2.770.0 ± 3.485.8 ± 1.97.7 ± 5.214.7 ± 3.827.0 ± 9.0
Grok self-refinement84.8 ± 2.080.4 ± 3.590.7 ± 0.214.7 ± 5.223.0 ± 0.040.0 ± 6.6
Claude self-refinement72.8 ± 0.867.6 ± 2.179.7 ± 1.45.7 ± 1.412.3 ± 7.616.0 ± 7.5
DeepSeek secondary corrector83.8 ± 1.979.2 ± 2.690.0 ± 1.09.3 ± 6.320.3 ± 3.834.7 ± 9.4
Grok secondary corrector86.1 ± 0.683.9 ± 0.589.0 ± 0.910.0 ± 6.623.0 ± 13.133.7 ± 11.2
Claude secondary corrector76.3 ± 2.169.6 ± 2.885.3 ± 1.95.7 ± 3.815.0 ± 2.520.0 ± 6.6
USP SFT82.9 ± 2.381.8 ± 3.984.3 ± 1.58.7 ± 5.729.7 ± 12.325.0 ± 6.6
USP SFT+RLCC83.9 ± 1.181.5 ± 0.287.2 ± 2.56.0 ± 8.627.0 ± 10.823.0 ± 15.5
DeepSeek + CoverSim89.0 ± 1.287.0 ± 2.291.8 ± 1.520.0 ± 5.040.3 ± 8.043.3 ± 13.7
Grok + CoverSim90.5 ± 0.989.8 ± 1.091.5 ± 0.818.3 ± 3.837.7 ± 10.039.7 ± 5.7
Claude + CoverSim84.0 ± 1.782.2 ± 3.186.4 ± 0.411.0 ± 6.629.0 ± 11.426.0 ± 9.0

Agent results: mean ± 95% CI (three runs; %)

AgentOriginal Acc@AllOriginal Acc@CoveredOriginal coverageCoverSim Acc@AllCoverSim Acc@CoveredCoverSim coverage
DeepSeek Flash99.3 ± 1.4100.0 ± 0.07.7 ± 2.980.3 ± 10.091.3 ± 19.318.0 ± 5.0
GPT-5-mini96.7 ± 1.4100.0 ± 0.010.3 ± 1.457.3 ± 12.769.4 ± 12.016.0 ± 8.6
GPT-596.7 ± 3.892.1 ± 19.111.7 ± 3.872.7 ± 7.691.7 ± 5.015.7 ± 7.6
Grok 4.395.7 ± 1.495.2 ± 20.59.0 ± 4.373.0 ± 6.679.1 ± 27.715.3 ± 7.2
DeepSeek V4 Pro97.7 ± 2.9100.0 ± 0.07.7 ± 2.975.7 ± 11.776.3 ± 5.717.0 ± 4.3
Claude Sonnet 4.690.3 ± 3.889.2 ± 25.19.7 ± 3.870.3 ± 3.880.6 ± 17.015.7 ± 3.8

Q1 - Could the covered subset affect agent comparisons?

We agree that Acc@Covered is conditional on the tests that the simulator executes conclusively. This is why we report it alongside coverage: together, they show agent performance on valid tests and how much of the intended test set was exercised. One can also inspect the covered profiles to understand the scope of user characteristics for which the reported accuracy is valid. A capable simulator should faithfully execute the intended profiles across the agents being evaluated. In practice, however, a test may be conclusive with one agent and inconclusive with another because the resulting conversations differ. We will clarify this caveat and interpret differences in Acc@Covered alongside coverage and uncertainty, rather than treating rankings from different covered subsets as definitive.

Reviewer_rE79OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that, although LLM-based user simulators are widely used in evaluating agent performance, the fidelity of the simulators is questionable and the coverage of situations is limited, and thus the evaluation cannot actually reflect the ranking of agent performance. The authors propose CoverSim, which first optimizes the prompt of a judge that decides whether the simulator faithfully exhibits the provided persona, and then uses that judge as the signal to optimize the simulator’s own prompt. The paper then shows a large gap between the optimized and the original simulator in evaluating agent performance, where the new simulator can detect the differences between agents more precisely.

Critical strengths

The paper identifies a real failure in simulator-based evaluation and quantifies it. With the standard SimulatorArena prompt, full-profile test coverage is only 6–12%. Consequently, 90% of the intended goal-profile conditions are not exercised, leading to the falsely high accuracy of the agents under original simulators. The paper clearly divides the outcome into pass/fail/inconclusive, giving a clean way to name and measure this.

The proposed method is cheap and effective. CoverSim leaves model weights untouched and needs only a few profiles and optimization steps, so it can apply to both open- and closed-source simulators with no training infrastructure. The improvement is significant in number: Full-profile coverage roughly triples for every prompting simulator tested.

Critical weaknesses

Conclusions drawn from the covered subset rest on <30 conversations with a single run. Each configuration is evaluated on 100 conversations with no repetitions and no confidence intervals anywhere in the paper, and coverage after optimization is 24–31% (Table 2), so the Acc@Covered metric is only measured on insufficient tasks. The 5×5 grid in §5.4 is also executed once per cell and suffers from high variance.

The simulator is only evaluated on the metric it was optimized for. CoverSim minimizes incoherent verdicts from the profile-consistency analyzer, but no evaluation or human study shows the trained simulator is more human-like, or only fit the judge. A second judge is needed to evaluate whether training improves behavior.

Base models are not held fixed, which makes the comparison less meaningful. USP is a weight-trained Llama-3-8B simulator, while CoverSim is built on frontier models. This difference makes the conclusion that “RL-based weight training is no better than direct prompt optimization” less convincing. Moreover, §5.1 notes that the simulator is deepseek-V4.1-flash for math tutoring but deepseek-v4-flash for document creation. Unifying the models would strengthen the conclusions.

Questions for authors

What changed between the original and optimized task-completion prompt on document creation? The accuracy goes from 44 to 76 (Table 3), while every other analyzer moves by only 1–4 points, so this looks less like an adjustment and more like the original prompt measuring the wrong thing. Could the authors show what parts of the prompt change before and after optimization?

See Weaknesses 1 and 2: The conclusion now rests on a limited number of samples and a single metric. If the authors can provide more statistical evidence (e.g., confidence intervals, multiple seeds) and more evaluation dimensions (e.g., a human study, a different model as judge), the conclusion will be strengthened, and I would be willing to raise my score.

5 comments
AuthorsOpenReview ↗

Author's Response 1

We thank the reviewer for recognizing the importance of reliable simulator-based evaluation, the distinction between conclusive and inconclusive tests, and the practicality of CoverSim. We address the reviewer’s concerns and question below.

C1 - Conclusions drawn from the covered subset rest on fewer than 30 conversations with a single run

We thank the reviewer for raising this concern. We repeated the math-tutoring evaluation three times and added 95% confidence intervals. CoverSim improves full-profile coherence for all simulator LLMs in every run. On covered tests, GPT-5 achieves 91.7 ± 5.0% accuracy versus 69.4 ± 12.0% for GPT-5-mini under CoverSim, supporting the observed difference across repeated evaluations. We repeated the coverage-grid experiment as well. Under the same budget of three attempts per profile, CoverSim consistently covers more cells than the original simulator in each run. This strengthens the grid’s role as a proof of concept for inspecting and improving coverage.

Experiment details.

For the main evaluation, each run contains 100 conversations per simulator setting. Results are means ± half-width of the 95% confidence interval across three runs (Student’s t). Full-profile coverage increases from 7.7 ± 2.9% to 20.0 ± 5.0% for DeepSeek, 11.0 ± 11.4% to 18.3 ± 3.8% for Grok, and 4.0 ± 2.5% to 11.0 ± 6.6% for Claude. Under CoverSim, conclusive tests account for 15.7 ± 7.6% of GPT-5’s conversations and 16.0 ± 8.6% of GPT-5-mini’s. The agent accuracy estimates remain based on relatively small covered subsets. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. The full results are in the tables below.

Simulator results: mean ± 95% CI (three runs)

ModelFull coherenceIS coherenceWS coherenceFull coverageIS coverageWS coverage
DeepSeek original81.7 ± 1.176.0 ± 1.089.2 ± 2.07.7 ± 2.914.7 ± 3.833.0 ± 9.0
Grok original86.1 ± 1.484.3 ± 2.088.4 ± 1.511.0 ± 11.427.3 ± 5.732.0 ± 13.8
Claude original72.5 ± 2.166.1 ± 2.581.2 ± 1.84.0 ± 2.511.0 ± 7.510.7 ± 3.8
DeepSeek self-refinement76.8 ± 2.770.0 ± 3.485.8 ± 1.97.7 ± 5.214.7 ± 3.827.0 ± 9.0
Grok self-refinement84.8 ± 2.080.4 ± 3.590.7 ± 0.214.7 ± 5.223.0 ± 0.040.0 ± 6.6
Claude self-refinement72.8 ± 0.867.6 ± 2.179.7 ± 1.45.7 ± 1.412.3 ± 7.616.0 ± 7.5
DeepSeek secondary corrector83.8 ± 1.979.2 ± 2.690.0 ± 1.09.3 ± 6.320.3 ± 3.834.7 ± 9.4
Grok secondary corrector86.1 ± 0.683.9 ± 0.589.0 ± 0.910.0 ± 6.623.0 ± 13.133.7 ± 11.2
Claude secondary corrector76.3 ± 2.169.6 ± 2.885.3 ± 1.95.7 ± 3.815.0 ± 2.520.0 ± 6.6
USP SFT82.9 ± 2.381.8 ± 3.984.3 ± 1.58.7 ± 5.729.7 ± 12.325.0 ± 6.6
USP SFT+RLCC83.9 ± 1.181.5 ± 0.287.2 ± 2.56.0 ± 8.627.0 ± 10.823.0 ± 15.5
DeepSeek + CoverSim89.0 ± 1.287.0 ± 2.291.8 ± 1.520.0 ± 5.040.3 ± 8.043.3 ± 13.7
Grok + CoverSim90.5 ± 0.989.8 ± 1.091.5 ± 0.818.3 ± 3.837.7 ± 10.039.7 ± 5.7
Claude + CoverSim84.0 ± 1.782.2 ± 3.186.4 ± 0.411.0 ± 6.629.0 ± 11.426.0 ± 9.0

Agent results: mean ± 95% CI (three runs; %)

AgentOriginal Acc@AllOriginal Acc@CoveredOriginal coverageCoverSim Acc@AllCoverSim Acc@CoveredCoverSim coverage
DeepSeek Flash99.3 ± 1.4100.0 ± 0.07.7 ± 2.980.3 ± 10.091.3 ± 19.318.0 ± 5.0
GPT-5-mini96.7 ± 1.4100.0 ± 0.010.3 ± 1.457.3 ± 12.769.4 ± 12.016.0 ± 8.6
GPT-596.7 ± 3.892.1 ± 19.111.7 ± 3.872.7 ± 7.691.7 ± 5.015.7 ± 7.6
Grok 4.395.7 ± 1.495.2 ± 20.59.0 ± 4.373.0 ± 6.679.1 ± 27.715.3 ± 7.2
DeepSeek V4 Pro97.7 ± 2.9100.0 ± 0.07.7 ± 2.975.7 ± 11.776.3 ± 5.717.0 ± 4.3
Claude Sonnet 4.690.3 ± 3.889.2 ± 25.19.7 ± 3.870.3 ± 3.880.6 ± 17.015.7 ± 3.8

On the 5×5 grid, the original simulator covers 12, 10, and 12 cells in the three runs, while CoverSim covers 19, 18, and 20. Across the three attempts, at least one is conclusive for 15/25 cells with the original simulator and 23/25 with CoverSim. We used a different grid after replacing the DeepSeek model snapshot and will update the paper’s figure to show the new runs.

Grid results: conclusive cells out of 25

Simulator promptRun 1Run 2Run 3Mean ± SDAt least onceAll threeNone
Original12101211.3 ± 1.215610
CoverSim19182019.0 ± 1.023142
AuthorsOpenReview ↗

Response 2

C2 - The simulator is only evaluated on the metric it was optimized for

We thank the reviewer for raising this concern. Our additional evaluations show that the coverage gains are not specific to the optimization oracle. To study this, we added two evaluations that were not used during simulator optimization. The first measures the same testing metrics with a different LLM judge, to check whether the gains depend on the original judge. The second measures human-likeness, a different property altogether, to check whether improvements extend beyond the optimization metric. The independent judge confirms coverage improvements across all three simulators; for example, DeepSeek’s full-profile coverage stays high under both judges (18% / 34%). CoverSim also improves human-likeness across all three simulators: for DeepSeek, the judge’s accuracy in distinguishing simulated from real conversations drops from 65.5% to 48.5% (chance is 50%). Notably, improved human-likeness emerges without directly optimizing for it. Together, these results provide evidence against overfitting to either the original judge or the optimization metric: the gains extend to a different judge and a different evaluation criterion.

Experiment details.

For coverage, we evaluated the same 100 conversations per simulator with the original DeepSeek analyzer and an independent Grok analyzer. Under the DeepSeek / Grok analyzers, full-profile coverage increased from 9% / 18% to 18% / 34% for DeepSeek, 10% / 20% to 20% / 27% for Grok, and 4% / 3% to 12% / 17% for Claude. Our coverage improvements remain valid on the independent Grok analyzer, which is not used during the GEPA optimization, proving we are not overfitting to the judge. The coverage numbers from the Grok analyzer are higher because it seems to be biased toward longer conversations.

Independent-analyzer results (DeepSeek / Grok; %; 100 conversations per simulator)

ModelFull coherenceIS coherenceWS coherenceFull coverageIS coverageWS coverageTurns
DeepSeek original82.1 / 83.376.4 / 78.189.8 / 90.29 / 1813 / 3636 / 466.3
Grok original85.6 / 86.283.4 / 83.588.6 / 89.810 / 2026 / 4331 / 466.8
Claude original72.5 / 73.766.2 / 65.780.8 / 84.34 / 38 / 1911 / 216.2
DeepSeek self-refinement75.8 / 75.468.9 / 65.684.9 / 88.56 / 1513 / 2323 / 434.0
Grok self-refinement85.7 / 82.381.9 / 75.990.7 / 90.817 / 1623 / 2642 / 463.6
Claude self-refinement73.1 / 75.768.5 / 68.879.3 / 84.86 / 99 / 2416 / 303.8
DeepSeek secondary corrector83.1 / 86.978.2 / 83.689.6 / 91.37 / 2322 / 4533 / 466.6
Grok secondary corrector85.9 / 85.183.9 / 81.388.8 / 90.212 / 1929 / 3829 / 426.8
Claude secondary corrector76.5 / 77.769.3 / 70.386.0 / 87.56 / 1116 / 2422 / 336.4
USP SFT81.8 / 88.580.0 / 90.284.2 / 86.26 / 2024 / 6224 / 349.9
USP SFT+RLCC83.6 / 89.181.4 / 90.786.5 / 87.14 / 2822 / 6021 / 409.9
DeepSeek + CoverSim89.0 / 89.887.5 / 86.991.1 / 93.618 / 3438 / 5137 / 617.5
Grok + CoverSim90.7 / 90.889.9 / 90.491.8 / 91.420 / 2737 / 5641 / 497.4
Claude + CoverSim83.2 / 83.180.9 / 80.386.3 / 86.812 / 1730 / 3623 / 356.4

For human-likeness, we use SimulatorArena’s conversation-level Turing test. A DeepSeek judge identifies the real conversation from a real–simulated pair on the same problem; we evaluate 100 pairs per simulator in both orders. Judge accuracy changes from 65.5% to 48.5% for DeepSeek, 75.0% to 64.0% for Grok, and 59.5% to 53.0% for Claude (closer to 50% means better). The full results are in the tables above and below.

Human-likeness Turing test (100 pairs; both orders)

SimulatorOriginal judge accuracy p (%)Original |p − 50|Original turnsCoverSim judge accuracy p (%)CoverSim |p − 50|CoverSim turns
DeepSeek Flash65.515.55.648.51.56.8
Grok 4.375.025.05.964.014.06.0
Claude Haiku 4.559.59.55.753.03.05.2

The second judge provides an independent check, though the judges may share biases; the human-likeness test is an automated proxy rather than a human study.

AuthorsOpenReview ↗

Response 3

C3 - Base models are not held fixed

We thank the reviewer for this suggestion. We repeated the comparison using models from the same Llama-3-8B family. USP achieves higher absolute coherence than CoverSim, while CoverSim improves coherence without updating the model’s weights. The two approaches use different starting checkpoints and optimization strategies, which is important when interpreting this comparison. USP fine-tunes Llama-3-8B on dialogue examples and then applies reinforcement learning to improve profile consistency. CoverSim instead uses an off-the-shelf instruction-tuned model and optimizes only its prompt. USP reaches 82.9% full-profile coherence after SFT and 83.9% after SFT and RL, while CoverSim improves the prompted model from 60.1% to 65.8%. Thus, most of USP’s advantage over the prompted simulator is already present after SFT. These results are encouraging for CoverSim: coverage-guided simulation could also be combined with stronger optimization methods, such as SFT or RL, although this is outside the scope of the paper. We deliberately use prompt optimization to show that the methodology can improve simulators without access to model weights, including closed-source models available through an API.

Experiment details.

We evaluate each simulator over three runs of 100 conversations, using the same DeepSeek tutor and USP’s sampling settings. CoverSim improves coherence in every run, although full-profile coverage remains low for Llama-3-8B-Instruct, increasing from 0.0% to 1.3% on average. USP begins from Llama-3-8B, while our prompted simulator uses Llama-3-8B-Instruct, so the comparison uses the same model family and size but not identical starting weights. All experiments were evaluated using the new version of deepseek-flash (since the version used in the paper is no longer accessible via API), which is a stricter oracle for profile coherence and task completion, explaining the score difference. The full results are in the table below.

Llama-3-8B simulator results: mean ± 95% CI (three runs; %)

ModelFull coherenceIS coherenceWS coherenceFull coverageIS coverageWS coverage
USP (SFT)82.9 ± 2.381.8 ± 3.984.3 ± 1.58.7 ± 5.729.7 ± 12.325.0 ± 6.6
USP (SFT + RLCC)83.9 ± 1.181.5 ± 0.287.2 ± 2.56.0 ± 8.627.0 ± 10.823.0 ± 15.5
Δ (RLCC)+1.0−0.3+2.9−2.7−2.7−2.0
Llama-3-8B-Instruct60.1 ± 4.265.7 ± 3.452.7 ± 10.50.0 ± 0.05.3 ± 3.80.7 ± 1.4
Llama-3-8B-Instruct + CoverSim65.8 ± 2.170.9 ± 0.959.0 ± 5.51.3 ± 3.86.0 ± 5.03.7 ± 3.8
Δ (CoverSim)+5.7+5.2+6.3+1.3+0.7+3.0

For document creation, we report results with the earlier DeepSeek model because the newer model already follows profiles well with the original prompt, leaving less room for improvement; results with the newer model are also discussed in Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.