- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Provenance-Adjusted Evidence (PAE) for selecting updates in self-improving LLM systems. PAE tracks the observations behind feedback, removes duplicated evidence, and adjusts aggregation. With complete per-task scores, it leads to averaging paired task gains. Experiments show that repeated interpretations of the same observations can alter update decisions, and replacing native selectors with PAE improves performance across several math, coding, and instruction-following benchmarks.
Critical strengths
Clear identification of an evaluation issue. The paper clearly distinguishes repeated feedback from independent evidence and demonstrates that repeatedly interpreting the same observations can alter update-selection decisions.
Careful experimental design. The paper evaluates multiple optimizers under controlled budgets, making the selector comparisons relatively clean.
Clear presentation and figures. The paper is well organized, and the figures and tables clearly illustrate the motivation, method, and main experimental results.
Critical weaknesses
The main experiments do not validate the most important part of PAE. With complete per-task scores, PAE reduces exactly to simple root averaging. Therefore, the reported downstream gains mainly show the benefit of replacing the native selector with a root-mean acceptance rule, rather than the benefit of provenance-aware overlap correction.
Limited validation on complex tasks. The evaluation is dominated by relatively short-form math, coding, and instruction-following benchmarks. The paper lacks validation on more complex settings, such as long-horizon agentic tasks or harder competition-level math and coding, where overlapping feedback provenance may be more realistic and consequential.
The gains may partly come from a more permissive acceptance policy. PAE accepts substantially more updates than the paired-test and PACE baselines while also showing higher false/harmful commit rates. Without comparisons at matched commit or risk levels, it is difficult to isolate the benefit of the proposed evidence aggregation rule.
Questions for authors
-
Since PAE reduces to root averaging in the main experiments, which results specifically demonstrate the benefit of provenance-aware overlap correction beyond a simpler root-mean selector?
-
Can the authors provide results on more complex tasks, such as long-horizon agentic benchmarks or harder competition-level math and coding, where feedback provenance and repeated evidence are likely to play a larger role?
-
Can the authors compare PAE with the baselines at matched commit rates or matched false/harmful-commit rates to determine whether the gains come from better evidence aggregation rather than a more permissive acceptance policy?
4 comments
Response to Reviewer KuH2 (2/2)
Q1. Which results demonstrate overlap benefits beyond a simple root-mean selector?
Section 4.3 proves exact equality between PAE and root mean under complete singleton feedback. The coverage experiment examines the contribution of joint judgments as singleton availability varies. On the same 100 MBPP candidate pairs:
| Singleton coverage | Equal-group HCR (%) | PAE HCR (%) | Equal-group net gain (pp) | PAE net gain (pp) |
|---|---|---|---|---|
| 0% | 21.05 | 18.42 | 0.484 | 0.542 |
| 25% | 20.51 | 15.79 | 0.511 | 0.566 |
| 50% | 18.42 | 13.16 | 0.521 | 0.582 |
| 75% | 15.79 | 11.11 | 0.550 | 0.587 |
| 100% | 13.51 | 11.11 | 0.556 | 0.587 |
At 100% coverage, PAE assigns zero weight to joint groups and equals root mean. Equal-group continues to weight joint groups. At 50%, using only available singletons gives pp, compared with PAE's pp:
| 50%-coverage selector | Accepted /100 | Beneficial / harmful accepts | Net gain per proposal (pp) |
|---|---|---|---|
| Available-singleton mean | 30 | 22 / 5 | 0.429 |
| Equal-group | 38 | 27 / 7 | 0.521 |
| PAE | 38 | 30 / 5 | 0.582 |
PAE uses the available joint evidence to improve recall and net gain over singleton-only averaging. At the same 38 accepted candidates, it also selects more beneficial and fewer harmful updates than Equal-group. These results identify the benefit of overlap adjustment under incomplete coverage.
Q2. Can you provide results on long-horizon agentic tasks?
The telecom experiment directly addresses your request. Qwen3-8B serves as both the task agent and user simulator, with greedy decoding shared across methods. One complete episode is one root. The 114 tasks are split into 12 proposal, 24 calibration, 48 acceptance, and 30 audit tasks. With , the acceptance split supports six fresh blocks.
| Feedback interface | Gate | Seed 0–4 successes, each /30 | Mean success (%) |
|---|---|---|---|
| Controlled joint | Dedup | 8, 7, 9, 8, 8 | 26.67 |
| Controlled joint | Equal-group | 8, 8, 9, 8, 9 | 28.00 |
| Controlled joint | PAE | 9, 9, 10, 9, 9 | 30.67 |
| Official complete score | S1 control selector | 9, 8, 10, 9, 9 | 30.00 |
| Official complete score | PAE = Root mean | 9, 9, 10, 9, 10 | 31.33 |
PAE achieves four additional successes over Equal-group across the five runs, which reuse the same 30 audit tasks. Under complete official scoring, PAE also achieves a higher mean success rate than the S1 outer-selector control. These results describe the stated 30-task audit subset and Qwen3-8B simulator configuration. They demonstrate improvements in multi-turn tool tasks under both feedback interfaces.
Q3. What happens at matched commit rates or matched false/harmful-commit rates?
Both controls are available. First, at the same 30/100 commits on identical joint-feedback candidates:
| Task | Equal-group beneficial / harmful | PAE beneficial / harmful | Equal-group / PAE HCR (%) |
|---|---|---|---|
| MBPP | 23 / 4 | 25 / 3 | 13.33 / 10.00 |
| Telecom | 21 / 6 | 23 / 4 | 20.00 / 13.33 |
Second, at a common 10% empirical HCR, threshold-mixture curves yield:
| Joint-feedback task | Gate | Beneficial recall (%) | FCR (%) | Net gain per proposal (pp) |
|---|---|---|---|---|
| MBPP | Equal-group | 37.78 | 19.47 | 0.361 |
| MBPP | PAE | 57.78 | 16.43 | 0.536 |
| Telecom | Equal-group | 22.50 | 19.00 | 0.330 |
| Telecom | PAE | 42.50 | 19.47 | 0.630 |
The 10% value is the empirical HCR operating point on these test curves. A separate calibration experiment selects and freezes thresholds before test evaluation:
| Task | Gate | Test accepts /100 | Test HCR (%) | Test beneficial recall (%) |
|---|---|---|---|---|
| MBPP | Equal-group | 21 | 9.52 | 37.78 |
| MBPP | PAE | 31 | 9.68 | 57.78 |
| Telecom | Equal-group | 11 | 9.09 | 22.50 |
| Telecom | PAE | 21 | 9.52 | 42.50 |
Calibration yields closely matched observed HCRs, with PAE achieving 20 percentage points higher beneficial recall in both domains. The exact-count and exact-HCR comparisons reach the same conclusion: PAE improves the observed selection tradeoff at controlled operating points.
Thank you for providing these additional experiments and details. I keep my original score.
Thank you
Thank you for taking the time to review our additional experiments and clarifications. We greatly appreciate your positive assessment and support for our work.
Response to Reviewer KuH2 (1/2)
Thank you for taking the time to review our work and for your constructive suggestions on task complexity and acceptance quality. Below, we respond to each point with the relevant experiments and controls.
Response. PAE accounts for the source observations shared by multiple judgments. With complete singleton feedback, this gives exact root averaging. Incomplete feedback activates the overlap adjustment. In a controlled interface with 50% singleton coverage, fixing acceptance at 30/100 yields:
The corresponding five-seed optimization experiment gives:
PAE selects more beneficial and fewer harmful updates at the same acceptance count, then achieves higher final scores in closed-loop optimization. Together, these comparisons demonstrate the contribution of cross-group overlap adjustment beyond identical-ancestry grouping.
Response. Your question concerns challenging math and coding problems as well as multi-turn agentic tasks. Table 2 reports results on the following harder benchmarks:
AIME and LiveCodeBench are transfer evaluations. The τ²-bench [1] telecom experiment extends the evaluation to multi-turn tool use:
This experiment uses five seeds, a 30-task held-out subset, and six update rounds. With PAE, the median episode contains 26 agent–user messages and 12 tool executions. Each complete episode forms one root. The higher success rate therefore extends the evidence to tasks requiring sustained interaction and tool use.
[1] τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment.
Response. At exactly 30 accepted candidates out of 100, PAE selects more beneficial updates and achieves a higher net gain in both direct-score banks. This controls the acceptance volume you identify:
Fixed-n ranks by one-sided test support. PACE uses support from the shared full root sequence at λ=0.5. The tables compare ranking quality at matched acceptance counts.
The theory in Section 4.5 considers both accepting worse candidates and rejecting better ones. PACE [1] controls per-candidate false-acceptance probability under its conditional null. Here, HCR measures the harmful fraction among accepted candidates. PACE's α=0.05 guarantee applies to its original test, and the matched-count comparison evaluates a different operating point. Its zero-commit result in Table 11 has HCR = NA. The matched-count results directly support a selection-quality contribution.
[1] PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents.