- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DEFT, an automatic prompt optimization method for open-ended factual generation. It decomposes model responses and reference answers into atomic claims to identify omissions and contradictions. These errors are generalized into content-invariant patterns and clustered into recurring error types. An optimizer LLM then proposes corrective directives, which are adopted based on a combined coverage and faithfulness objective with paired-bootstrap stability checking. The method is evaluated on K-QA, TruthfulQA, and FActScore across four language-model backbones. The authors report improvements in factual coverage and reliability, with additional experiments examining error generalization, clustering, directive adoption, and optimization efficiency.
Critical strengths
-
The paper addresses a meaningful problem in automatic prompt optimization for open-ended generation. Using atomic claims to identify missing and contradictory information provides more specific feedback than response-level scores. The overall framework is reasonably designed and connects error detection, generalization, clustering, and prompt refinement.
-
The experimental evaluation is relatively comprehensive. The authors evaluate DEFT on three benchmarks and four language models, with additional ablations and analyses of data efficiency, optimizer selection, and computational cost. The improvements in K-QA comprehensiveness and hallucination metrics are encouraging.
Critical weaknesses
-
The methodological novelty seems relatively limited. Atomic claim decomposition has been widely studied in factuality evaluation [1], while ETGPO already uses recurring error types to guide prompt optimization [2]. DEFT mainly combines these ideas for open-ended generation. More importantly, the paper does not directly compare claim-level feedback with holistic response feedback under the same optimization framework. The current ablations focus on generalization, clustering, and bootstrap selection, making it difficult to isolate the contribution of claim-level feedback.
-
The paper does not clearly demonstrate whether the generated directives actually reduce the specific claim-level errors identified during optimization. The experiments mainly report aggregate benchmark metrics, without systematically analyzing targeted error types before and after optimization. In addition, the proposed faithfulness metric only penalizes claims explicitly contradicted by the reference answer. Factually incorrect claims that are not mentioned in the reference may still receive a high faithfulness score. It is unclear how reliably this signal captures hallucinations.
-
The generalization and validation experiments are not fully convincing. Candidate directives are generated and evaluated on the same optimization batch, without an independent validation set. Bootstrap resampling does not fully address potential overfitting from this adaptive selection process. In addition, DEFT optimizes separate prompts for different datasets. Although the method aims to generate content-invariant directives, the paper does not evaluate whether these prompts transfer across datasets without further optimization. Similar cross-dataset transfer experiments have already been considered in GPO [3].
-
The additional complexity and computational cost are not sufficiently justified by the performance gains. DEFT introduces multiple stages of claim decomposition, error detection, clustering, and directive selection, but ultimately optimizes a single answer-generation prompt. On TruthfulQA, it consumes approximately 1.68M tokens during optimization, compared with 0.12M for ETGPO. Some improvements over strong baselines are also small. For example, DEFT improves K-QA comprehensiveness from 0.728 to 0.738 and TruthfulQA T×I from 0.776 to 0.787 compared with APE. GPO also achieves a higher FActScore (60.1 vs. 45.0), although with a lower response rate. Without compute-matched comparisons, it is difficult to assess whether the additional complexity provides sufficient benefits.
[1] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023.
[2] Error Taxonomy-Guided Prompt Optimization. arXiv 2026.
[3] Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers. AAAI 2025.
Questions for authors
-
What is the main difference between DEFT and ETGPO? Can the authors directly compare claim-level feedback with holistic feedback under the same optimization framework?
-
Do the generated directives actually reduce the specific claim-level errors identified during optimization? How are incorrect claims that are not explicitly contradicted by the reference handled?
-
Why are candidate directives generated and evaluated on the same batch? Can the optimized prompts generalize across datasets without further optimization?
-
Would DEFT maintain its advantage over existing baselines under matched computational budgets, given its substantially higher token consumption?
4 comments
W2) Targeted Error Reduction Analysis and Unsupported Claim Handling
Thank you for this important comment. We conducted an additional claim-level analysis to examine whether the directives generated by DEFT reduce the specific errors they target.
1. Targeted Error Reduction Analysis
We analyzed the effect of newly adopted directives across successive optimization rounds, following DEFT's cumulative prompt optimization process. Using Qwen3.5-4B on the TruthfulQA test set, we compared responses generated with the prompt before adding a directive and after adding it, while retaining all previously adopted directives.
At round , we identified the targeted error types () using the directives adopted in each round of the original DEFT optimization run and their corresponding error clusters. We then applied the same error extraction procedure to the test-set responses and counted the targeted errors before directive adoption () and the remaining errors after adoption ().
| Round () | Targeted Errors () | Remaining Errors () | Resolved Errors () | Reduction (%) |
|---|---|---|---|---|
| 1 | 49 | 38 | 11 | 22.4 |
| 2 | 363 | 234 | 129 | 35.5 |
| 3 | 229 | 140 | 89 | 38.9 |
| 4 | 47 | 24 | 23 | 48.9 |
Resolved Errors denotes the number of previously identified targeted errors that were resolved after adding the directive. Reduction (%) is the proportion of these errors relative to the targeted errors before adding the directive.
Across all four rounds, the number of targeted errors consistently decreased after the corresponding directive was added. Each round resolved a portion of the targeted errors. For example, in round 2, targeted errors decreased from 363 to 234, with 129 errors (35.5%) resolved. These findings provide additional evidence that DEFT's adopted directives can reduce the specific claim-level errors they target.
2. Unsupported Claim Handling
We agree that reference-based contradiction detection may miss factually incorrect claims that are not explicitly contradicted by the gold reference. Consider the following example:
- Question: Name three Impressionist painters.
- Gold reference: Monet, Renoir, and Degas are Impressionist painters.
- Generated response: Monet, Pissarro, and Leonardo da Vinci are Impressionist painters.
In this example, classifying Leonardo da Vinci as an Impressionist painter is factually incorrect. However, since the gold reference only lists valid Impressionist painters without explicitly addressing Leonardo da Vinci, DEFT may not identify this claim as a contradiction.
DEFT's faithfulness metric measures contradictions against the reference rather than comprehensive factual correctness. To address this limitation, DEFT could be extended with external knowledge verification, allowing incorrect claims beyond the reference to be identified and incorporated into the existing directive optimization process.
We will clarify this limitation and potential extension in the revised paper.
W3) Same Batch Overfitting and Insufficient Cross Dataset Generalization
Same Batch Overfitting
We agree that evaluating and selecting directives on the same optimization batch may introduce selection bias and overfitting. Paired bootstrap assesses the stability of improvements within the optimization batch but does not guarantee generalization.
To examine this point, we evaluated the prompt after every accepted optimization round on the TruthfulQA test set, which was not used during optimization. The final prompt was determined by the last accepted round, rather than selected based on test-set performance. The following table reports the round-wise results for Qwen3.5-4B.
| Round | Truth ↑ | Info ↑ | T×I ↑ |
|---|---|---|---|
| Vanilla | 77.1 | 95.7 | 73.8 |
| 1 | 73.7 | 94.1 | 70.6 |
| 2 | 79.4 | 94.4 | 75.2 |
| 3 | 79.9 | 94.9 | 76.4 |
| 4 | 83.2 | 96.2 | 80.2 |
Three of the four accepted optimization steps improved held-out T×I relative to the preceding prompt, and the final prompt outperformed the vanilla baseline. Although the improvement was not monotonic across rounds, these results indicate that gains observed during optimization can transfer to unseen test examples, providing evidence against the concern that DEFT's gains arise solely from overfitting to the optimization batch.
Cross-Dataset Generalization
We agree that cross-dataset transfer should be evaluated. We therefore conducted additional zero-shot evaluations on LongFact-Objects and GAMUT using prompts optimized on TruthfulQA, without further optimization on either benchmark. To provide a broader comparison, we also evaluated prompts from APE and ETGPO under the same settings.
| Method | LongFact F1 ↑ | GAMUT ↑ |
|---|---|---|
| Vanilla | 0.595 | 37.7 |
| DEFT | 0.597 ± 0.005 | 32.2 ± 5.6 |
| APE | 0.585 ± 0.014 | 33.7 ± 3.1 |
| ETGPO | 0.539 ± 0.063 | 26.3 ± 14.0 |
We evaluated 50 randomly sampled LongFact questions using GPT-4o-mini as the judge and a local Wikipedia BM25 retriever. For GAMUT, we evaluated the text-only subset using GPT-4o-mini. Because of our limited evaluation budget, the judge differs from those used in the original papers. However, all methods were evaluated with the same judge and data under identical conditions, ensuring a controlled comparison within this experiment.
No prompt-optimization method consistently improves over Vanilla across both out-of-distribution datasets, indicating that cross-dataset transfer remains a general challenge for prompt optimization. Nevertheless, DEFT does not exhibit a serious degradation. On LongFact, it preserves the Vanilla performance (0.595 → 0.597). On GAMUT, DEFT obtains a lower mean score than Vanilla (32.2 vs. 37.7), but the Vanilla score lies within one standard deviation of the DEFT mean (32.2 ± 5.6). Thus, the observed difference is within DEFT's variability and does not provide evidence of a dramatic performance collapse.
These results suggest that our statement in Section 3 that a directive “generalizes across questions and domains” was too broad. DEFT abstracts question-specific facts and entities into recurring error types, allowing directives to address similar errors across different questions. Our round-wise test-set analysis also provides evidence of generalization beyond the optimization batch. However, these properties do not guarantee that the resulting prompts will transfer to datasets with different error distributions and task requirements. We will revise the statement in Section 3.
W4) Justification of Computational Cost and Complexity
We agree that the original comparison did not fully account for differences in optimization cost. We first extend Appendix Table 10 to report external metric evaluator token usage alongside local token consumption and inference time.
Optimization-stage computational cost (Appendix Table 10).
| Method | Local Tokens (Input / Output) | Local Inference Time (s) | External Metric Evaluator Input Tokens |
|---|---|---|---|
| APE | 0.12M / 0.33M | 577 | 2.04M |
| GPO | 0.11M / 0.89M | 878 | 4.33M |
| PromptAgent | 0.06M / 0.14M | 224 | 0.80M |
| ETGPO | 0.06M / 0.06M | 115 | 0 |
| ZERA | 0.11M / 0.02M | 99 | 0 |
| DEFT | 1.19M / 0.49M | 339 | 0 |
DEFT processes more local tokens due to claim decomposition and error identification. More importantly, DEFT uses internal coverage and faithfulness signals computed by the optimizer LLM to select directives, without requiring external task-metric evaluation during optimization. In contrast, APE, GPO, and PromptAgent use dataset-specific evaluation metrics as optimization signals. For open-ended QA, evaluating generated responses often relies on LLM-based judges [1][2], and repeated evaluations can lead to substantial API costs, particularly when using more expensive judges. DEFT avoids this dependency during optimization.
Compute-matched comparison. For fair comparison of the performance–compute trade-off, we evaluated all methods under an approximately matched local token usage of about 1.7M on TruthfulQA using Qwen3.5-4B. We adjusted the number of candidates, iterations without changing the underlying algorithms.
| Method | T×I ↑ | Local Tokens (Input / Output) | Total Local Tokens | External Metric Evaluator Input Tokens |
|---|---|---|---|---|
| APE | 80.4 ± 3.5 | 0.51M / 1.12M | 1.63M | 7.18M |
| GPO | 74.9 ± 0.9 | 0.41M / 1.34M | 1.75M | 5.74M |
| PromptAgent | 74.2 ± 1.2 | 0.57M / 0.68M | 1.25M | 4.43M |
| ETGPO | 70.5 ± 5.1 | 0.74M / 0.49M | 1.23M | 0 |
| ZERA | 50.5 ± 9.2 | 1.33M / 0.18M | 1.51M | 0 |
| DEFT | 78.7 ± 1.3 | 1.19M / 0.49M | 1.68M | 0 |
Local Tokens (Input / Output) report optimization-stage input and output tokens from Qwen3.5-4B, and Total Local Tokens is their sum.
APE achieves the highest T×I (80.4 ± 3.5), followed by DEFT (78.7 ± 1.3). Thus, DEFT does not outperform all baselines under approximately matched local token budgets, but remains competitive without external metric evaluation calls during optimization.
We will update Appendix Table 10 and include the compute-matched results to clarify DEFT's computational trade-offs.
References
[1] Han et al. (2024). RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering. EMNLP 2024. https://aclanthology.org/2024.emnlp-main.249/
[2] Wei et al. (2024). Long-form Factuality in Large Language Models. NeurIPS 2024. https://arxiv.org/abs/2403.18802
We thank the reviewer for the detailed and constructive feedback. We conducted additional analyses and experiments to clarify DEFT's methodological contribution, effectiveness, generalization, and computational trade-offs.
W1) Methodological Novelty and Claim Level Feedback Ablation
Thank you for highlighting the need to clarify our distinction from ETGPO and directly evaluate the contribution of claim-level feedback. While both methods leverage recurring error patterns for prompt optimization, they differ in how error signals are constructed and incorporated into the optimization process.
Distinction from ETGPO. ETGPO analyzes failed reasoning traces to identify underlying failure modes, organizes them into a global error taxonomy, and generates guidance for frequently occurring error types. In contrast, DEFT compares atomic claims from reference and generated responses to identify omissions and contradictions, grounding its error types in explicit discrepancies between the response and the reference. This allows DEFT to identify specific deficiencies even in partially correct open-ended responses, without relying on an overall success/failure judgment.
Furthermore, whereas ETGPO constructs guidance from a global error taxonomy, DEFT proposes directives based on claim-level errors observed under the current prompt and selectively adopts them based on response-quality evaluations. It then re-examines errors under the updated prompt in subsequent optimization rounds. Thus, DEFT connects reference-grounded claim-level feedback to the proposal and evaluation of prompt revisions within an iterative optimization process.
In partially correct responses to open-ended questions, essential information may be omitted while other claims contradict the reference. While ETGPO identifies recurring error patterns from failed reasoning traces, DEFT directly compares atomic claims from reference and generated responses to distinguish omissions from contradictions. This enables DEFT to identify specific aspects of partially correct responses that require improvement and use them to guide prompt optimization without relying on an overall success or failure judgment.
Claim-level vs. holistic feedback ablation. We added a controlled ablation in which claim-level error extraction was replaced by holistic response-level feedback. In the holistic variant, the optimizer evaluates the response against the reference as a whole, without decomposing either into atomic claims. All subsequent stages, including error abstraction, clustering, directive generation, adoption, remain unchanged. We evaluated the variants on TruthfulQA using Qwen3.5-4B across three seeds (mean ± SD).
Full DEFT outperforms holistic feedback across all three seeds, achieving a 6.0 percentage point improvement in T×I. These results provide empirical evidence for the effectiveness of reference-grounded claim-level feedback within DEFT's optimization framework.