acceptodds
Under review as a conference paper at ICLR 2027

Claim-Level Feedback Signals for Prompt Optimization in Open-Ended Generation

Abstract

Optimizing prompts for frozen language models typically relies on scalar rewards or holistic feedback over entire responses. In open-ended generation, however, a response may contain many factual claims, making such coarse feedback insufficient to identify what should be corrected. We formulate prompt refinement around fine-grained, claim-level error signals. Our approach decomposes model responses and reference answers into atomic claims, identifies coverage gaps and contradictions, and abstracts these errors into content-invariant patterns that capture recurring failure modes across examples. The resulting patterns are clustered into error types from which corrective prompt directives are proposed. Each directive is adopted only when its improvement over the current prompt satisfies both a gain margin and a paired-bootstrap stability criterion. Across four language-model backbones and three open-ended factuality benchmarks—K-QA, TruthfulQA, and FActScore—the resulting prompts improve the balance between factual coverage and reliability, including higher comprehensiveness and lower hallucination on K-QA. These results show that decomposed error signals provide a more actionable objective for prompt optimization than treating an open-ended response as a single evaluation unit.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Automatic Prompt Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Zz6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEFT, an automatic prompt optimization method for open-ended factual generation. It decomposes model responses and reference answers into atomic claims to identify omissions and contradictions. These errors are generalized into content-invariant patterns and clustered into recurring error types. An optimizer LLM then proposes corrective directives, which are adopted based on a combined coverage and faithfulness objective with paired-bootstrap stability checking. The method is evaluated on K-QA, TruthfulQA, and FActScore across four language-model backbones. The authors report improvements in factual coverage and reliability, with additional experiments examining error generalization, clustering, directive adoption, and optimization efficiency.

Critical strengths
  1. The paper addresses a meaningful problem in automatic prompt optimization for open-ended generation. Using atomic claims to identify missing and contradictory information provides more specific feedback than response-level scores. The overall framework is reasonably designed and connects error detection, generalization, clustering, and prompt refinement.

  2. The experimental evaluation is relatively comprehensive. The authors evaluate DEFT on three benchmarks and four language models, with additional ablations and analyses of data efficiency, optimizer selection, and computational cost. The improvements in K-QA comprehensiveness and hallucination metrics are encouraging.

Critical weaknesses
  1. The methodological novelty seems relatively limited. Atomic claim decomposition has been widely studied in factuality evaluation [1], while ETGPO already uses recurring error types to guide prompt optimization [2]. DEFT mainly combines these ideas for open-ended generation. More importantly, the paper does not directly compare claim-level feedback with holistic response feedback under the same optimization framework. The current ablations focus on generalization, clustering, and bootstrap selection, making it difficult to isolate the contribution of claim-level feedback.

  2. The paper does not clearly demonstrate whether the generated directives actually reduce the specific claim-level errors identified during optimization. The experiments mainly report aggregate benchmark metrics, without systematically analyzing targeted error types before and after optimization. In addition, the proposed faithfulness metric only penalizes claims explicitly contradicted by the reference answer. Factually incorrect claims that are not mentioned in the reference may still receive a high faithfulness score. It is unclear how reliably this signal captures hallucinations.

  3. The generalization and validation experiments are not fully convincing. Candidate directives are generated and evaluated on the same optimization batch, without an independent validation set. Bootstrap resampling does not fully address potential overfitting from this adaptive selection process. In addition, DEFT optimizes separate prompts for different datasets. Although the method aims to generate content-invariant directives, the paper does not evaluate whether these prompts transfer across datasets without further optimization. Similar cross-dataset transfer experiments have already been considered in GPO [3].

  4. The additional complexity and computational cost are not sufficiently justified by the performance gains. DEFT introduces multiple stages of claim decomposition, error detection, clustering, and directive selection, but ultimately optimizes a single answer-generation prompt. On TruthfulQA, it consumes approximately 1.68M tokens during optimization, compared with 0.12M for ETGPO. Some improvements over strong baselines are also small. For example, DEFT improves K-QA comprehensiveness from 0.728 to 0.738 and TruthfulQA T×I from 0.776 to 0.787 compared with APE. GPO also achieves a higher FActScore (60.1 vs. 45.0), although with a lower response rate. Without compute-matched comparisons, it is difficult to assess whether the additional complexity provides sufficient benefits.

[1] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023.

[2] Error Taxonomy-Guided Prompt Optimization. arXiv 2026.

[3] Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers. AAAI 2025.

Questions for authors
  1. What is the main difference between DEFT and ETGPO? Can the authors directly compare claim-level feedback with holistic feedback under the same optimization framework?

  2. Do the generated directives actually reduce the specific claim-level errors identified during optimization? How are incorrect claims that are not explicitly contradicted by the reference handled?

  3. Why are candidate directives generated and evaluated on the same batch? Can the optimized prompts generalize across datasets without further optimization?

  4. Would DEFT maintain its advantage over existing baselines under matched computational budgets, given its substantially higher token consumption?

4 comments

We thank the reviewer for the detailed and constructive feedback. We conducted additional analyses and experiments to clarify DEFT's methodological contribution, effectiveness, generalization, and computational trade-offs.

W1) Methodological Novelty and Claim Level Feedback Ablation

Thank you for highlighting the need to clarify our distinction from ETGPO and directly evaluate the contribution of claim-level feedback. While both methods leverage recurring error patterns for prompt optimization, they differ in how error signals are constructed and incorporated into the optimization process.

Distinction from ETGPO. ETGPO analyzes failed reasoning traces to identify underlying failure modes, organizes them into a global error taxonomy, and generates guidance for frequently occurring error types. In contrast, DEFT compares atomic claims from reference and generated responses to identify omissions and contradictions, grounding its error types in explicit discrepancies between the response and the reference. This allows DEFT to identify specific deficiencies even in partially correct open-ended responses, without relying on an overall success/failure judgment.

Furthermore, whereas ETGPO constructs guidance from a global error taxonomy, DEFT proposes directives based on claim-level errors observed under the current prompt and selectively adopts them based on response-quality evaluations. It then re-examines errors under the updated prompt in subsequent optimization rounds. Thus, DEFT connects reference-grounded claim-level feedback to the proposal and evaluation of prompt revisions within an iterative optimization process.

In partially correct responses to open-ended questions, essential information may be omitted while other claims contradict the reference. While ETGPO identifies recurring error patterns from failed reasoning traces, DEFT directly compares atomic claims from reference and generated responses to distinguish omissions from contradictions. This enables DEFT to identify specific aspects of partially correct responses that require improvement and use them to guide prompt optimization without relying on an overall success or failure judgment.

Claim-level vs. holistic feedback ablation. We added a controlled ablation in which claim-level error extraction was replaced by holistic response-level feedback. In the holistic variant, the optimizer evaluates the response against the reference as a whole, without decomposing either into atomic claims. All subsequent stages, including error abstraction, clustering, directive generation, adoption, remain unchanged. We evaluated the variants on TruthfulQA using Qwen3.5-4B across three seeds (mean ± SD).

VariantTruth↑Info↑T×I↑
Vanilla77.1 ± 0.395.7 ± 0.573.8 ± 0.4
Holistic DEFT77.9 ± 0.893.9 ± 2.772.7 ± 2.0
DEFT81.9 ± 1.396.2 ± 0.178.7 ± 1.3

Full DEFT outperforms holistic feedback across all three seeds, achieving a 6.0 percentage point improvement in T×I. These results provide empirical evidence for the effectiveness of reference-grounded claim-level feedback within DEFT's optimization framework.

W2) Targeted Error Reduction Analysis and Unsupported Claim Handling

Thank you for this important comment. We conducted an additional claim-level analysis to examine whether the directives generated by DEFT reduce the specific errors they target.

1. Targeted Error Reduction Analysis

We analyzed the effect of newly adopted directives across successive optimization rounds, following DEFT's cumulative prompt optimization process. Using Qwen3.5-4B on the TruthfulQA test set, we compared responses generated with the prompt before adding a directive and after adding it, while retaining all previously adopted directives.

At round tt, we identified the targeted error types (TEtTE_{t}) using the directives adopted in each round of the original DEFT optimization run and their corresponding error clusters. We then applied the same error extraction procedure to the test-set responses and counted the targeted errors before directive adoption (TEtTE_{t}) and the remaining errors after adoption (REtRE_{t}).

Round (tt)Targeted Errors (TEtTE_{t})Remaining Errors (REtRE_{t})Resolved Errors (TEt−REtTE_{t} - RE_{t})Reduction (%)
149381122.4
236323412935.5
32291408938.9
447242348.9

Resolved Errors denotes the number of previously identified targeted errors that were resolved after adding the directive. Reduction (%) is the proportion of these errors relative to the targeted errors before adding the directive.

Across all four rounds, the number of targeted errors consistently decreased after the corresponding directive was added. Each round resolved a portion of the targeted errors. For example, in round 2, targeted errors decreased from 363 to 234, with 129 errors (35.5%) resolved. These findings provide additional evidence that DEFT's adopted directives can reduce the specific claim-level errors they target.

2. Unsupported Claim Handling

We agree that reference-based contradiction detection may miss factually incorrect claims that are not explicitly contradicted by the gold reference. Consider the following example:

  • Question: Name three Impressionist painters.
  • Gold reference: Monet, Renoir, and Degas are Impressionist painters.
  • Generated response: Monet, Pissarro, and Leonardo da Vinci are Impressionist painters.

In this example, classifying Leonardo da Vinci as an Impressionist painter is factually incorrect. However, since the gold reference only lists valid Impressionist painters without explicitly addressing Leonardo da Vinci, DEFT may not identify this claim as a contradiction.

DEFT's faithfulness metric measures contradictions against the reference rather than comprehensive factual correctness. To address this limitation, DEFT could be extended with external knowledge verification, allowing incorrect claims beyond the reference to be identified and incorporated into the existing directive optimization process.

We will clarify this limitation and potential extension in the revised paper.

W3) Same Batch Overfitting and Insufficient Cross Dataset Generalization

Same Batch Overfitting

We agree that evaluating and selecting directives on the same optimization batch may introduce selection bias and overfitting. Paired bootstrap assesses the stability of improvements within the optimization batch but does not guarantee generalization.

To examine this point, we evaluated the prompt after every accepted optimization round on the TruthfulQA test set, which was not used during optimization. The final prompt was determined by the last accepted round, rather than selected based on test-set performance. The following table reports the round-wise results for Qwen3.5-4B.

RoundTruth ↑Info ↑T×I ↑
Vanilla77.195.773.8
173.794.170.6
279.494.475.2
379.994.976.4
483.296.280.2

Three of the four accepted optimization steps improved held-out T×I relative to the preceding prompt, and the final prompt outperformed the vanilla baseline. Although the improvement was not monotonic across rounds, these results indicate that gains observed during optimization can transfer to unseen test examples, providing evidence against the concern that DEFT's gains arise solely from overfitting to the optimization batch.

Cross-Dataset Generalization

We agree that cross-dataset transfer should be evaluated. We therefore conducted additional zero-shot evaluations on LongFact-Objects and GAMUT using prompts optimized on TruthfulQA, without further optimization on either benchmark. To provide a broader comparison, we also evaluated prompts from APE and ETGPO under the same settings.

MethodLongFact F1 ↑GAMUT ↑
Vanilla0.59537.7
DEFT0.597 ± 0.00532.2 ± 5.6
APE0.585 ± 0.01433.7 ± 3.1
ETGPO0.539 ± 0.06326.3 ± 14.0

We evaluated 50 randomly sampled LongFact questions using GPT-4o-mini as the judge and a local Wikipedia BM25 retriever. For GAMUT, we evaluated the text-only subset using GPT-4o-mini. Because of our limited evaluation budget, the judge differs from those used in the original papers. However, all methods were evaluated with the same judge and data under identical conditions, ensuring a controlled comparison within this experiment.

No prompt-optimization method consistently improves over Vanilla across both out-of-distribution datasets, indicating that cross-dataset transfer remains a general challenge for prompt optimization. Nevertheless, DEFT does not exhibit a serious degradation. On LongFact, it preserves the Vanilla performance (0.595 → 0.597). On GAMUT, DEFT obtains a lower mean score than Vanilla (32.2 vs. 37.7), but the Vanilla score lies within one standard deviation of the DEFT mean (32.2 ± 5.6). Thus, the observed difference is within DEFT's variability and does not provide evidence of a dramatic performance collapse.

These results suggest that our statement in Section 3 that a directive “generalizes across questions and domains” was too broad. DEFT abstracts question-specific facts and entities into recurring error types, allowing directives to address similar errors across different questions. Our round-wise test-set analysis also provides evidence of generalization beyond the optimization batch. However, these properties do not guarantee that the resulting prompts will transfer to datasets with different error distributions and task requirements. We will revise the statement in Section 3.

W4) Justification of Computational Cost and Complexity

We agree that the original comparison did not fully account for differences in optimization cost. We first extend Appendix Table 10 to report external metric evaluator token usage alongside local token consumption and inference time.

Optimization-stage computational cost (Appendix Table 10).

MethodLocal Tokens (Input / Output)Local Inference Time (s)External Metric Evaluator Input Tokens
APE0.12M / 0.33M5772.04M
GPO0.11M / 0.89M8784.33M
PromptAgent0.06M / 0.14M2240.80M
ETGPO0.06M / 0.06M1150
ZERA0.11M / 0.02M990
DEFT1.19M / 0.49M3390

DEFT processes more local tokens due to claim decomposition and error identification. More importantly, DEFT uses internal coverage and faithfulness signals computed by the optimizer LLM to select directives, without requiring external task-metric evaluation during optimization. In contrast, APE, GPO, and PromptAgent use dataset-specific evaluation metrics as optimization signals. For open-ended QA, evaluating generated responses often relies on LLM-based judges [1][2], and repeated evaluations can lead to substantial API costs, particularly when using more expensive judges. DEFT avoids this dependency during optimization.

Compute-matched comparison. For fair comparison of the performance–compute trade-off, we evaluated all methods under an approximately matched local token usage of about 1.7M on TruthfulQA using Qwen3.5-4B. We adjusted the number of candidates, iterations without changing the underlying algorithms.

MethodT×I ↑Local Tokens (Input / Output)Total Local TokensExternal Metric Evaluator Input Tokens
APE80.4 ± 3.50.51M / 1.12M1.63M7.18M
GPO74.9 ± 0.90.41M / 1.34M1.75M5.74M
PromptAgent74.2 ± 1.20.57M / 0.68M1.25M4.43M
ETGPO70.5 ± 5.10.74M / 0.49M1.23M0
ZERA50.5 ± 9.21.33M / 0.18M1.51M0
DEFT78.7 ± 1.31.19M / 0.49M1.68M0

Local Tokens (Input / Output) report optimization-stage input and output tokens from Qwen3.5-4B, and Total Local Tokens is their sum.

APE achieves the highest T×I (80.4 ± 3.5), followed by DEFT (78.7 ± 1.3). Thus, DEFT does not outperform all baselines under approximately matched local token budgets, but remains competitive without external metric evaluation calls during optimization.

We will update Appendix Table 10 and include the compute-matched results to clarify DEFT's computational trade-offs.

References

[1] Han et al. (2024). RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering. EMNLP 2024. https://aclanthology.org/2024.emnlp-main.249/

[2] Wei et al. (2024). Long-form Factuality in Large Language Models. NeurIPS 2024. https://arxiv.org/abs/2403.18802

Reviewer_VafJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a automatic prompt optimization (APO) approach to improve the factuality of a frozen LLM on long-form open-ended generation tasks. It relies on decompose-search-verify pipelines such as FactScore and VeriScore as supervision signals for the factuality of long-form responses, which break a response into atomic claims before verifying each claim against external evidence (e.g. Wikipedia). Such methods have been used extensively in previous work for finetuning LLMs to improve their factuality, but this is the first work to apply this approach to APO.

Critical strengths
  • This paper applies an established approach to a new problem, automatic prompt optimization, which is a practical use case as finetuning is not possible in many cases.

  • The proposed approach is sound, and borrows useful ideas from existing work such as building error types, which results in a sensible and effective method.

Critical weaknesses
  • While the empirical results appear to be strong, it is only tested on in-distribution examples. Training data is drawn from each test set, which is insufficient to establish the generalization and real-world performance of the proposed approach. Adding more out-of-distribution test sets that are not trained on would make the results much more convincing.

  • Most, if not all, technical components of this work are not new. It mainly applies the known recipe in factuality finetuning to APO, while integrating several techniques in existing APO literature. While sensible and well-articulated, the technical contribution of this work is a bit thin.

  • This paper follows FactScore to decompose an open-ended response into atomic claims, and compare that with a gold response to derive the supervision signals. Recent work (e.g. https://arxiv.org/abs/2607.19322) has highlighted multiple issues with this approach.

    • First, the answer space of open-ended questions is much more complex than what a flat list of binary atomic facts can express. For question "name a few impressionist paintings", the answer space is an open-ended list with many possible correct answers, none of which is strictly required. Similarly, for "how can I make creme brulee", a good answer must not only identify the key steps, but also the correct order of those steps.
    • In addition, this work uses a single gold answer as reference for grading a given response, which is limiting for open-ended questions. As in the previous example "name a few impressionist paintings", the gold answer may name 3-5 famous impressionist paintings, but there are many other acceptable alternatives. Checking against a single gold answer can neither accurately capture misses nor contradictions. Failing to mention a particular painting in the gold answer does not necessarily mean that the response should be penalized. Similarly, mentioning an artwork not present in the gold answer does not mean that it has to be a hallucination.
Questions for authors

Q1: Can you apply DEFT to additional open-ended generation benchmarks (without training on those), such as LongFact, GAMUT, etc.?

4 comments

We thank the reviewer for the detailed feedback. We address three aspects of DEFT's evaluation and methodology.

W1 & Q1) Out of Distribution Generalization

Thank you for this suggestion. To evaluate cross-dataset generalization, we conducted additional zero-shot evaluations on LongFact-Objects and GAMUT using prompts optimized on TruthfulQA, without further optimization on either benchmark. To provide a broader comparison of cross-dataset transfer, we also evaluated prompts from APE and ETGPO under the same settings.

MethodLongFact F1 ↑GAMUT ↑
Vanilla0.59537.7
DEFT0.597 ± 0.00532.2 ± 5.6
APE0.585 ± 0.01433.7 ± 3.1
ETGPO0.539 ± 0.06326.3 ± 14.0

We evaluated 50 randomly sampled LongFact questions using GPT-4o-mini as the judge and a local Wikipedia BM25 retriever. For GAMUT, we evaluated the text-only subset using GPT-4o-mini. Because of our limited evaluation budget, the judge differs from those used in the original papers. However, all methods were evaluated with the same judge and data under identical conditions, ensuring a controlled comparison within this experiment.

No prompt-optimization method consistently improves over Vanilla across both out-of-distribution datasets, indicating that cross-dataset transfer remains a general challenge for prompt optimization. Nevertheless, DEFT does not exhibit a serious degradation. On LongFact, it preserves the Vanilla performance (0.595 → 0.597). On GAMUT, DEFT obtains a lower mean score than Vanilla (32.2 vs. 37.7), but the Vanilla score lies within one standard deviation of the DEFT mean (32.2 ± 5.6). Thus, the observed difference is within DEFT's variability and does not provide evidence of a dramatic performance collapse. We will include these results and explicitly discuss this limitation in the revised paper.

W2) Technical Novelty

Thank you for your comment. While DEFT builds on established techniques, an important challenge remains in open-ended QA. Responses can be partially correct, omitting essential information while introducing contradictions. A single correctness score does not explicitly distinguish these failure modes, making it difficult to determine how a prompt should be improved.

Existing APO approaches such as ZERA and APE rely on aggregate performance scores or general error feedback, which may provide limited guidance on what specifically needs to be improved in partially correct open-ended responses. DEFT addresses this challenge by explicitly distinguishing omissions from reference contradictions through atomic claim comparison and identifying recurring error patterns to propose targeted directives. Importantly, these error types require different corrections, and addressing one may worsen the other. Rather than incorporating every recurring error pattern, DEFT selectively adopts directives based on a joint coverage and faithfulness objective. This design distinguishes DEFT from existing approaches by connecting explicit claim-level error signals to both directive generation and performance-guided adoption, enabling more targeted optimization of partially correct responses.

Our contribution lies not in developing new decomposition or clustering algorithms, but in formulating actionable feedback for partially correct open-ended responses and integrating it with performance-guided directive adoption. We will clarify this contribution in the revised paper.

W3) Atomic Claim Decomposition and Single Gold Reference

Thank you for this opinion. We clarify how DEFT handles different types of claims through reference-based comparison, provide additional evidence on the usefulness of omission feedback, and discuss possible extensions for answer validation and procedural dependencies.

1. Multiple Valid Answers and Reference Dependence

Consider the following example:

  • Question: Name at least three Impressionist painters.
  • Gold reference: Monet, Renoir, and Degas are Impressionist painters.
  • Generated response: Monet, Pissarro, and Leonardo da Vinci are Impressionist painters. Renoir is not an Impressionist painter.

After atomic claim decomposition, we can distinguish the following cases:

CategoryExampleDEFT's Treatment
Correct ClaimMonetNo error
MissedDegasMiss
ContradictedRenoirCont
Unverified Correct ClaimPissarro (valid Impressionist painter)Ignored since not identified as an error
Undetected Factual ErrorLeonardo da Vinci (not an Impressionist painter)Ignored since not identified as an error
Absent from both Reference and ResponseMary Cassatt (valid Impressionist painter)Outside the scope of comparison
Absent from both Reference and ResponseJean-François Millet (not an Impressionist painter)Outside the scope of comparison

The table illustrates how DEFT handles different types of claims based on the gold reference and generated response. Category describes each claim's status relative to the reference and response, while DEFT's Treatment indicates how the claim is handled within the DEFT algorithm.

DEFT identifies missed claims when reference claims are not covered by the response and contradicted when generated claims conflict with the reference. In this example, Degas is identified as missed while Renoir is identified as a contradicted.

Importantly, DEFT does not classify a claim as a contradiction merely because it is absent from the reference. Pissarro is a valid alternative and is not automatically penalized as a hallucination. However, DEFT does not explicitly verify alternative correct answers against external knowledge. Likewise, a factual error such as classifying Leonardo da Vinci as an Impressionist painter may remain undetected when the reference provides insufficient evidence. Therefore, DEFT ignores both categories without applying a penalty.

Information absent from both the reference and generated response falls outside the current claim comparison. For example, Mary Cassatt is another valid Impressionist painter, while Jean-François Millet is not. Neither appears in the reference or response, so neither is considered within the DEFT algorithm. Incorporating external knowledge could extend DEFT's ability to assess such information beyond the reference and response.

DEFT aims to improve open-ended QA by leveraging missed and contradicted signals obtained from comparing reference and generated responses. However, as the reviewer mentioned, the answer space of open-ended questions is highly complex. Future work could extend DEFT's algorithm by incorporating external knowledge and structured answer requirements such as rubrics, allowing more comprehensive validation of alternative answers, factual correctness, and relationships between claims while preserving its core optimization procedure.

2. Procedural Dependencies

We acknowledge that independent atomic claims may not fully capture procedural dependencies. In the reviewer's creme brulee example, identifying individual preparation steps does not guarantee their correct ordering. A possible extension is to represent ordering constraints as relational claims and check whether generated responses violate these constraints.

This extension would require additional inference and depend on the accuracy of extracting procedural relationships. Its effectiveness remains untested. We will clarify these limitations and potential extensions in the revised paper.

Thank you for your response.

I appreciate the inclusion of cross-dataset generalization results. Unfortunately, the result confirms my concern that the proposed approach fails to improve over even the vanilla baseline when the training examples do not come from the test set. Furthermore, even compared within prompt optimization approaches, DEFT does not consistently achieve superior accuracy, despite being more costly. I will maintain my original score.

Thank you for your follow-up and for taking the time to consider our additional results.

We agree that DEFT does not consistently outperform Vanilla in the cross-dataset setting. We observed similar results for the other prompt optimization methods we evaluated, including APE and ETGPO, suggesting that zero-shot transfer across datasets remains a broader challenge for current prompt optimization methods. Our intention was to report these findings transparently, rather than to claim improvements in cross-dataset transfer.

Your comment also helped us recognize that our statement in Section 3, that a directive "generalizes across questions and domains," was broader than what our experimental evidence supports. We have therefore revised this statement to more precisely reflect the scope of generalization considered in our work. (page 5 line 232-233)

We appreciate your careful review and respect your assessment.

Reviewer_iw5mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a method for open-ended question answerwing. The authors conduct experiement by comparing their method to several baselines. Ablation studies were also done to evaluate how different hyperparamters affect the performance.

Critical strengths
  1. Method is clear and well-motivated. The algorithmic pipeline is easy to understand.
  2. The method clearly win over the baselines. Empirical results are strong.
Critical weaknesses

GPO is not in Table 2 but only in Table 1. Why this is the case? It seems like GPO's performance is decent in Table 1. Similarly, APE is not in Table 2.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the constructive feedback. To strengthen the cross-model comparison, we additionally evaluated APE and GPO on GLM4-9B. DEFT achieves the best K-QA hallucination and TruthfulQA T×I scores. These results provide a more comprehensive comparison across optimization methods and model families.

W1) APE and GPO results in Table 2

Thank you for pointing out the omission of APE (ICLR 2023) and GPO (AAAI 2025) from Table 2.

Table 2 was primarily designed to evaluate DEFT across different model families. Due to space constraints, we included the two most recent baselines, ZERA (EMNLP 2025) and ETGPO (arXiv 2026), while providing a broader baseline comparison in Table 1.

To answer your question, we additionally evaluated APE and GPO on GLM4-9B, representing a different model family from Qwen3.5-4B used in Table 1. The results are summarized below.

MethodK-QA Comp↑K-QA Hall↓TruthfulQA T×I↑FActScore↑
Vanilla0.637 ± 0.0050.614 ± 0.0630.768 ± 0.00337.6 ± 0.2
APE0.645 ± 0.0200.554 ± 0.0540.718 ± 0.02042.8 ± 0.6
GPO0.615 ± 0.0760.508 ± 0.0560.727 ± 0.00252.0 ± 3.3
DEFT0.639 ± 0.0050.373 ± 0.0710.777 ± 0.01246.2 ± 7.1

On GLM4-9B, DEFT achieves the best K-QA hallucination and TruthfulQA T×I scores, while ranking second in K-QA completeness and FActScore. Our analysis of adopted directives suggests that DEFT primarily targeted contradictions on K-QA, leading to greater improvements in hallucination reduction than completeness. On FActScore, its emphasis on verification may have limited gains in factual precision compared with directives encouraging more informative and well-supported content.

We will add these results and clarify the baseline selection criterion for Table 2 in the revised paper.

Thank you for your rebuttal and adding these results. I will keep my rating for acceptance.

Thank you for your constructive feedback. We sincerely appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Results

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback. DEFT optimizes prompts for open-ended generation by identifying claim-level omissions and contradictions and transforming recurring error types into targeted directives. We appreciate the reviewers' recognition of the method's clarity and motivation (iw5m), practical relevance (Vafj, Zz6X), and comprehensive evaluation (Zz6X).

We summarize below how our additional experiments and analyses address the main points raised during the review.

  1. Contribution of Claim-Level Feedback (Vafj, Zz6X): We conducted a controlled ablation replacing claim-level feedback with holistic feedback while keeping all other optimization stages unchanged. The results showed that DEFT achieved better performance on TruthfulQA.

  2. Targeted Error Reduction (Zz6X): In a held-out analysis, adopted directives reduced their corresponding targeted error types, suggesting that DEFT's directives can address the errors they target.

  3. Generalization (Vafj, Zz6X): Round-wise evaluation on the held-out test set showed that optimization gains can carry over to unseen examples. Cross-dataset experiments on LongFact and GAMUT showed mixed results, with similar limitations observed across other APO baselines, including APE and ETGPO.

  4. Computational Cost (Zz6X): Under approximately matched local token budgets, DEFT remained competitive with strong baselines without relying on external task-specific metric evaluation during optimization.

  5. Additional Baselines and Clarifications (iw5m, Vafj, Zz6X): We provided additional APE and GPO results on GLM4-9B, clarified the methodological distinction from ETGPO, and discussed the limitations of single-reference comparison and possible extensions.

Overall, these additional experiments and analyses further support DEFT's central contribution of using actionable claim-level error feedback for targeted prompt optimization in open-ended generation, while also helping clarify the scope of its effectiveness.

Detailed results and discussions are available in our individual responses to the reviewers.

We thank the Area Chair and reviewers again for their time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.