acceptodds
Under review as a conference paper at ICLR 2027

Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs

26 views

Abstract

Closed-loop revision is increasingly adopted in large language model applications to iteratively reduce mismatches between generated outputs (e.g., a paragraph) and user targets (e.g., required paragraph length). However, revision failures may stem from incomplete feedback or models failing to act effectively on correct feedback, making it difficult to systematically assess the revision capabilities of LLMs. In this work, we propose a fixed-budget revision protocol in which deterministic verifiers report all remaining violations across three constraint families: exact-length, lexical, and compositional. This holds feedback correctness and completeness fixed, allowing us to isolate and evaluate model-side closed-loop revision. We conduct experiments on 19 mainstream open-source and closed-source models, including Llama 3.1 8B and GPT-5.6 Sol, and observe large cross-model and cross-constraint differences in final joint success, with controller-level mean final joint success ranging from 17.4% to 99.8%. Substantial cross-model gaps persist when the same initial draft is used. In controlled experiments, we find reproducible model-specific differences in how exact feedback is translated into revisions, while post-training and model scale reshape these responses without consistently bringing them closer to exact correction. Interestingly, across all constraint families, failure trajectories often repeat earlier outputs, and prior recurrence is associated with substantially lower subsequent recoverability. Finally, matched-state interventions across constraint families show that removing earlier dialogue, with the current draft and feedback fixed, changes recurrence escape without reliably improving final success; effects depend on the model, task, and trigger-state composition. Overall, exact feedback makes revision errors observable, but does not make the closed loop reliable. Code and data are available anonymously.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, iterative revision, feedback-guided revision, exact feedback, closed-loop evaluation, output recurrence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fvyDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies closed-loop revision in LLM applications. To reduce the influence of incomplete or unreliable feedback, the authors propose a fixed-budget revision protocol to evaluate LLMs' revision ability in a controlled setting. They design three types of constraints, exact-length, lexical, and compositional, and evaluate a range of LLMs on these tasks. The results show that model performance still varies substantially even when the feedback is accurate and complete.

Critical strengths

Evaluating the closed-loop revision ability of LLMs appears to be a worthwhile direction for further study. In my view, it is valuable to distinguish the quality of feedback from the model’s ability to revise its output based on that feedback.

Critical weaknesses
  1. The three constraint families considered in this paper are quite limited and mainly focus on surface-level linguistic properties of the responses. They do not evaluate the quality of the underlying answers, nor do they examine how answer quality changes as models improve constraint satisfaction. It is therefore possible that a revision satisfies the formal constraints while making the answer itself less accurate, relevant, or useful. The paper does not capture this potential trade-off.
  2. Although these three constraint families allow the authors to provide fully deterministic and correct feedback, the feedback is highly regularized and task-specific. Such feedback also appears relatively easy to optimize for directly during post-training, for example through reinforcement-learning-style objectives. This raises a concern about whether the benchmark is measuring a model’s general revision ability, or mainly its ability to follow and optimize for a small set of predefined constraints.
Questions for authors
  1. How does task performance change across revision rounds as constraint satisfaction improves? In particular, it would be useful to examine whether better constraint satisfaction is accompanied by improvements, degradation, or trade-offs in the quality of the underlying task response.
  2. How would targeted reinforcement learning on these tasks affect model performance, both in terms of constraint satisfaction and task quality? It would also be interesting to train a model on one specific constraint family and evaluate whether the resulting behavior transfers to the other two constraint families.
2 comments

Thank you for recognizing the value of studying revision ability separately from feedback quality. Your questions prompted us to examine what happens to the text as it meets the constraints and what transfers when a model trains on just one constraint family.

W1. Answer quality during revision

“It is therefore possible that a revision satisfies the formal constraints while making the answer itself less accurate, relevant, or useful.”

Q1

“How does task performance change across revision rounds as constraint satisfaction improves? In particular, it would be useful to examine whether better constraint satisfaction is accompanied by improvements, degradation, or trade-offs in the quality of the underlying task response.”

On exact-length tasks, all 12 local baselines showed their largest loss of initial-draft entities on the first edit. Across 4,515 model–task pairs with identifiable entities in the initial draft, model-averaged retention fell to 96.0% at that step and was 95.8% at round 8. Over the same rounds, cumulative success across all 5,760 pairs rose from 3.2% to 19.8%. Our Official Comment “A round-by-round audit of content retention and sentence consistency” gives the full trajectory.

To assess the writing itself, GPT-6 Luna and Claude Haiku 5.5 compared drafts against the original writing request with the added constraints hidden. We counted a preference only when both judges chose the same draft in both presentation orders. Across 8,910 exact-length trajectories, they jointly favored the initial draft and the first edit almost equally often. On those same trajectories, they favored the initial draft in 1,031 cases and the final draft in 312. Details are in our official comment "Two-judge comparisons of writing quality across drafts".

The clearest trade-off appeared on the edit that first satisfied every lexical constraint. Across 1,308 such repairs, both judges preferred the preceding draft in 38.8% of cases and the passing draft in 9.4%. For Claude Opus 5, the direction was reversed. Across its 220 repairs, both preferred the passing draft in 28.6% of cases and its predecessor in 16.8%. The lexical and compositional judgments assessed clarity, coherence, and completeness of meaning. The judges had no independent factual reference for those tasks, so the evidence speaks most directly to how the text reads.

W2. Targeted training and cross-family transfer

“This raises a concern about whether the benchmark is measuring a model’s general revision ability, or mainly its ability to follow and optimize for a small set of predefined constraints.”

Q2

“How would targeted reinforcement learning on these tasks affect model performance, both in terms of constraint satisfaction and task quality? It would also be interesting to train a model on one specific constraint family and evaluate whether the resulting behavior transfers to the other two constraint families.”

Interestingly, we had considered RL as a solution but didn't finish it, because just as the experiment we added proved, closed-loop revision might not the best solution for these constrained tasks considering its cost. Before submission, we ran a 256-task REINFORCE++ pilot with Qwen3-8B on exact-length revision. Following your suggestion, we extended that run to 1,024 training tasks and tested both untrained families. Exact-length final success rose from 9.6% to 26.9%. Targeted training helped, although only about one in four tasks passed by the end of revision. It's far from perfect.

What transferred most consistently was a change in editing behavior. At each successive checkpoint, early revisions less often repeated an earlier failed answer, including on both untrained families. Final success rose modestly on lexical constraints (35.4% to 38.1%) and more on compositional constraints (69.4% to 80.2%). On the 480 compositional tasks, initial successes rose only from 284 to 288, while cases first solved during revision nearly doubled, from 49 to 97. This single run shows transfer across tasks with exact feedback. Our Official Comment “Exact-length post-training and cross-family transfer” reports the training setup and checkpoint results.

We also compared the two checkpoints’ final answers on the same exact-length tasks. Each generated its own initial draft. After training, final answers retained a smaller share of their initial-draft entities (−2.53 percentage points across 363 applicable pairs). Coverage of entities in the shared reference text changed little (+0.07 points across 436 pairs). RL raised success in all three families, yet both judges more often preferred baseline final answers in both orders. On lexical tasks, they favored baseline in 81 of 480 pairs and RL in 36.

Thank you for suggesting these experiments. We would include the round-by-round quality results, the transfer experiment, and their limitations in the revised manuscript.

Thank you for the thoughtful response and additional experiments, and apologies for my delayed reply. I appreciate the new findings. However, the W1 results also lend some support to my concerns about the trade-off between constraint satisfaction and answer quality, while the generalizability of revision ability remains uncertain. I will therefore maintain my current rating.

Reviewer_HtwSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of LLM closed-loop revision under exact feedback. The authors introduce a deterministic verification framework covering three constraint families: exact-length, lexical, and compositional constraints. They evaluate 19 model configurations over multiple revision rounds and find substantial differences in revision performance, even when starting from identical drafts.

Critical strengths
  1. The paper provides a systematic evaluation of closed-loop revision across 19 models and three constraint families. The use of deterministic verifiers helps separate feedback quality from the model's ability to incorporate feedback.

  2. The analysis of revision trajectories is useful, particularly the fixed-draft comparisons and history-reset experiments. These experiments provide additional insights into model-specific correction behavior beyond final success rates.

Critical weaknesses
  1. The novelty and significance of the paper seem limited given the existing literature on feedback-driven self-correction. Prior work such as DeCRIM (EMNLP 2024), Stechly et al. (ICLR 2025), RefineBench (ICLR 2026), and Meeseeks (ACL 2026) has already investigated iterative revision under external feedback, including multi-constraint instruction following and reasoning tasks. Although this paper introduces more controlled analyses of feedback-to-action responses and recurrence, it is unclear whether these findings provide sufficiently new insights beyond existing studies.

  2. The experimental setting is too narrow to support the broader motivation of reliable closed-loop revision. The tasks mainly involve exact-length, lexical, and compositional text constraints, many of which are deterministic and can be addressed more directly through rule-based editing or constrained decoding without relying on LLMs. This raises questions about the practical relevance of using repeated LLM revisions for such tasks. Furthermore, stronger models already achieve nearly perfect success, while the paper does not compare against simpler non-LLM correction methods. It remains unclear whether the observed failures reveal general limitations of closed-loop revision or mainly reflect difficulties with basic instruction following.

Questions for authors
  1. How does this work advance beyond existing feedback-driven correction benchmarks such as Meeseeks and RefineBench, particularly in terms of new findings about revision failures?

  2. How does the proposed closed-loop revision protocol compare with simpler alternatives, such as fresh regeneration, constrained decoding, or deterministic editing, in terms of success and computational cost?

4 comments
AuthorsOpenReview ↗

Response to HtwS (1/2): Novelty and revision failures

Thank you for highlighting the fixed-draft and history experiments.

W1. Novelty and significance

“The novelty and significance of the paper seem limited given the existing literature on feedback-driven self-correction.”

Q1

“How does this work advance beyond existing feedback-driven correction benchmarks such as Meeseeks and RefineBench, particularly in terms of new findings about revision failures?”

We appreciate the opportunity to clarify our aim. Our title, Exact Feedback Is Not Control, highlights the distinction we study in closed-loop revision. We do not propose a state-of-the-art solver for constrained writing. These tasks let us test the limits of the model’s own revision ability under controlled conditions. Every target is attainable, and after each attempt an oracle reports every remaining constraint violation exactly. Can the model use that feedback to revise its own draft until it satisfies every constraint? When it falls short, where does the repair break down?

  • RefineBench shows how much models can gain from guidance. GPT-4.1 judges each checklist item on every turn. In self-refinement the model decides what to change without seeing those judgments. In full guided refinement it receives every item GPT-4.1 marked No, so the completeness of the diagnosis depends on the judge.
  • DeCRIM uses GPT-4-Turbo as its RealInstruct oracle. On IFEval, its rule-based oracle flags failed constraints, but the revision prompt simply repeats their wording. For a missed length target, the model must work out how many words to add or remove (App. G.2, Box 5). The authors also report unachievable constraints (§5.3), so a failed repair can reflect an infeasible request. Every exact-length case in our study has a verified solution, and the report gives the exact word-count correction and all missing required phrases.
  • Meeseeks documents word-count overshoot even when models receive the actual count. An LLM directly judges constraints without rule labels (§3.1–3.3). Even for rule-labeled constraints, a model may determine which portion of the response the script checks. We check the submitted draft directly.
  • Stechly et al. provide a comparable sound verifier that can report every conflicting edge in graph coloring. Their tasks require solving arithmetic, coloring, and planning problems. Our text tasks isolate a more basic editing ability, such as adding or removing a specified number of words. In their fixed-candidate probes, the model identifies errors in a proposed solution. In our fixed-draft probes, the model receives an exact diagnosis and we measure the edit it actually makes. Their history ablation removes the current candidate and its critique along with earlier attempts. We keep the draft and diagnosis fixed to isolate how earlier dialogue affects the next edit.

What do we learn about revision failures? Even with oracle feedback, models can miss the requested edit size or direction and get stuck repeating a draft. Checkpoints show distinct tendencies to overshoot, undershoot, or leave the word count unchanged. These tendencies recur on unseen texts and help predict edit sizes in independent revision runs. Instruction tuning can reshape a shared failure pattern in different ways, and larger checkpoints do not consistently make more accurate corrections (§4.2).

For example, the Llama and Qwen 8B base models both barely shortened their drafts when explicitly asked to delete words. After instruction tuning, their responses diverged. On held-out requests to add or delete words, Llama edited in the requested direction 90% of the time, up from 31% for its base model. Qwen made fewer edits in the wrong direction, yet left the word count unchanged on nearly half of requests, up from just over a third (§4.2).

Across all three task families, roughly four in five to nine in ten failed trajectories returned to a previous draft verbatim. Many ended by repeating one draft or cycling through a small set of drafts (§5.1). On matched recurrent states, changing only the earlier dialogue could either help or hinder the model’s next attempt to escape repetition. Even large changes in escape rates usually produced much smaller changes in final success (§5.2).

The GLM pilot illustrates this gap. On selected recurrent states from lexical and compositional tasks, removing earlier dialogue raised the chance of escaping repetition on the next edit from one in ten to two in three. Yet after history removal, only one in twenty continuations ultimately satisfied all constraints (§5.2). Systems increasingly rely on models to act on feedback. We reveal that even for simple and feasible tasks, an oracle diagnosis won't necessarily make a model to make correct revision. The value of our research lies in its empirical contribution: we analyze this failure mode of the model and investigate potential interventions.

AuthorsOpenReview ↗

Response to HtwS (2/2): Task scope and simpler alternatives

W2. Scope and practical relevance

“The experimental setting is too narrow to support the broader motivation of reliable closed-loop revision.”

“It remains unclear whether the observed failures reveal general limitations of closed-loop revision or mainly reflect difficulties with basic instruction following.”

Scope. Missing a word-count target on a first attempt could reflect basic instruction following. We chose attainable targets with exact feedback so that we could ask a more specific question: once the model is told exactly what is wrong with its draft, can it make the requested edit? On an exact-length task, we know precisely how many words the model needs to add or remove, so we can measure both the direction and the size of its response. We can also hold the draft and diagnosis fixed and vary only the earlier dialogue. The same model can then respond differently to the same correction (§5.2). This shows that revision history contributes to the observed behavior even when the current instruction is unchanged. How far this finding extends to open-ended revision remains an empirical question.

Strong models. Near-perfect final success can conceal how much repair was needed. On exact-length tasks, GPT-5.6 Sol and Claude Opus 5 rarely satisfy all constraints on their first draft. Both reach near-perfect success by the end of the revision loop (Table 9). On the exact-length panel, Llama 3.1 8B achieved higher final success than three frontier API models one might expect to perform better: DeepSeek V4 Flash, GLM-5.2, and Qwen3.7 Plus (Table 1). GLM-5.2 nevertheless solved nearly every compositional case. A label like “strong” obscures how differently models revise their drafts and the trade-offs they face between meeting constraints and preserving writing quality (see our official comments).

“ the paper does not compare against simpler non-LLM correction methods.”

Q2

“How does the proposed closed-loop revision protocol compare with simpler alternatives, such as fresh regeneration, constrained decoding, or deterministic editing, in terms of success and computational cost?”

We appreciate the question. Success rate and cost matter when choosing a system for constrained writing. Our contribution in Exact Feedback Is Not Control is empirical. We study how reliably a model can revise its own draft when every remaining violation is identified, and where its attempts break down. We do not propose a new method or claim state-of-the-art performance on these tasks.

Fresh regeneration tests whether the model can produce a compliant answer on another attempt from scratch. Constrained decoding tests whether enforcing a constraint during generation yields a compliant answer. Deterministic editing tests whether a rule can correct an existing draft directly. Even if such a rule repaired every draft at negligible cost, two questions would remain. Given a complete and accurate diagnosis, can an otherwise capable model carry out a simple edit we might ask of a human writer, such as removing ten words while keeping every required phrase? When it falls short, where does the repair break down? The alternatives you suggest may satisfy the constraints, but their success alone tells us little about where the model reaches the limits of its own revision ability.

Thank you to the authors for the detailed rebuttal and additional experiments.

The response helps me better understand the distinction between evaluating a model's ability to act on exact feedback and developing an effective constraint-solving system. However, my main concerns about the novelty and significance of the work remain. While the authors clarify how their experimental setup differs from prior studies, I am still not fully convinced that the findings provide sufficiently substantial new insights beyond existing work on feedback-guided revision.

Overall, I acknowledge that the rebuttal has addressed some of my concerns and improved my appreciation of the paper's empirical contributions. I will therefore raise my score to 2.

AuthorsOpenReview ↗

Thank you and synthesis of core conceptual insights

Thank you very much for re-evaluating our work, engaging with our rebuttal, and raising your score to 2.

Regarding your remaining concern on whether our findings offer sufficiently substantial new insights beyond prior work, we would like to distill the three core takeaways that existing iterative-revision studies could not isolate:

  1. Separating Diagnosis Bottlenecks from Actuation Bottlenecks (Observability ≠\neq Control): Prior works primarily evaluate end-to-end pass rates where incomplete feedback or infeasible targets can confound failure, or promote oracle feedback as a solution for their systems. By guaranteeing feasible targets and providing exact verifier diagnostics, we isolate actuation fidelity—showing that models exhibit stable, checkpoint-specific edit biases (under-correction, directional asymmetry, and near-zero action) that generalize to unseen texts and do not monotonically improve with model scale or instruction tuning (§4.2 & Appendix.E).

  2. Decoupling Output Recurrence from Closed-Loop Completion: Aggregate success curves conflate getting stuck in history with being unable to execute the edit. Our trajectory and matched-state history interventions (§5.1–§5.2) reveal that the vast majority of failed runs repeat earlier drafts verbatim, yet breaking the loop is not enough: removing prior dialogue can substantially alter next-step escape rates (helping or hindering depending on the model and task) while barely moving final task completion.

  3. Implications for Verifier-Guided Systems: An increasing number of studies have found that more comprehensive feedback within a closed-loop system can effectively improve completion rates. Based on these studies, many modern agentic and reasoning pipelines implicitly embrace the assumption that pairing an LLM with exact feedback and extra revision turns ensures convergence. Our results demonstrate that exact feedback provides observability, not control (§6)—verifier precision alone cannot guarantee closed-loop reliability without addressing model-level action calibration and dialogue-state dynamics.

We will sharpen the Introduction and Discussion in the final version to foreground these takeaways and their broader implications for closed-loop LLM systems. Thank you again for your constructive feedback.

Reviewer_rSP2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can reliably revise constrained text using complete and accurate feedback from deterministic verifiers. It evaluates 19 model configurations across three constraint families, comprising 1,440 test cases. Through fixed-draft comparisons, controlled correction experiments, and dialogue-history interventions, the authors analyze revision performance and failure patterns. The results reveal substantial differences across models and tasks, systematic editing behaviors, and frequent repetition of incorrect outputs. The findings suggest that exact feedback alone does not guarantee successful revision.

Critical strengths
  1. Well-motivated problem and careful experimental design. The paper addresses an important problem in iterative LLM revision and provides a systematic evaluation of 19 open-weight and API-based models across three constraint families. The use of deterministic verifiers ensures correct and complete feedback, allowing the authors to isolate model-side revision behavior. The fixed-draft comparisons further strengthen the findings by showing that differences in revision performance persist even when models start from identical responses.
  2. Informative findings on LLM revision behavior and failure patterns. The paper provides useful insights into how different models respond to exact feedback, identifying systematic editing patterns that replicate on unseen texts. The analysis of output repetition and dialogue-history interventions is particularly useful, showing that escaping repeated outputs or improving individual revisions does not necessarily lead to successful task completion. These findings can offer a better understanding of the limitations of iterative LLM revision beyond final success rates.
Critical weaknesses
  1. Clarity and Writing Quality: Overall, the paper needs substantial revision to improve clarity and readability. Several sentences are too long or difficult to understand, and some technical terms are not clearly defined. I recommend revising the entire manuscript, particularly the abstract, main contributions, and discussion sections. E.g, "As Figure 6 shows, next step escape increases by 42.2 percentage points for Llama and 49.2 for GLM." Define "next step escape" before using it.
  2. Assessment of content preservation: Exact-length success requires the target word count and specified words or phrases, but does not establish that the response preserves the intended meaning or overall quality. The reported results therefore measure constraint satisfaction rather than revision quality more broadly.
  3. Limited generality. The evaluation focuses on automatically checkable text constraints, such as word counts and required words. These tasks allow complete, exact feedback, whereas factual correction and open-ended revision often involve ambiguous or incomplete feedback. The findings are informative within the evaluated setting, but their applicability to these broader revision tasks remains untested.
  4. Methodological novelty seems limited. Prior work has already investigated LLM revision failures and differences in feedback incorporation across models. While this paper provides a more systematic analysis through controlled experiments and detailed failure characterization, it does not introduce a new revision method or a solution to the identified limitations. Its main contribution is therefore a deeper empirical understanding of existing problems rather than a methodological advancement.
Questions for authors

First, revise the paper and improve the clarity and quality of writing. Also, I suggest making the distinction between satisfying the checked constraints and preserving the response’s meaning and quality more explicit. Finally, evaluation on broader revision tasks can be provided if feasible, otherwise it can be discussed and suggested as future work.

3 comments

Thank you for your careful reading and for recognizing the value of the fixed-draft comparisons and the failure analysis. We also appreciate your pointing to specific passages where a reader needs a definition before seeing the result.

W1. Clarity and writing quality

“Several sentences are too long or difficult to understand, and some technical terms are not clearly defined.”

“Define "next step escape" before using it.”

In revising the manuscript, we would define technical terms at first use and split sentences that combine distinct findings, with particular attention to the abstract, main contributions, and Discussion. The examples below illustrate how we would revise passages throughout the paper.

In §5.2 we would define the term before Figure 6: “A next-step escape occurs when the model’s next draft differs from every draft it produced before the intervention.” In §4.2, “zero-action” would mean that the word count stays the same even if the wording changes.

Abstract (original). “We conduct experiments on 19 mainstream open-source and closed-source models, including Llama 3.1 8B and GPT-5.6 Sol, and observe large cross-model and cross-constraint differences in final joint success, with controller-level mean final joint success ranging from 17.4% to 99.8%.”

Proposed. “We evaluated 19 model configurations. Final success varied across models and constraint families. Mean success across families ranged from 17.4% to 99.8% across configurations.”

Discussion (original). “At the trajectory level, recurrent outputs mark states with lower subsequent recoverability, while matched-history interventions show that changing retained dialogue can strongly alter local continuation behavior without reliably improving final completion.”

Proposed. “Once a model repeated an earlier draft, it was less likely to satisfy all constraints in the remaining rounds. When we held the current draft and feedback fixed, changing the earlier dialogue could alter the model’s next edit. This did not consistently improve final success.”

W2. Assessment of content preservation

“Exact-length success requires the target word count and specified words or phrases, but does not establish that the response preserves the intended meaning or overall quality.”

We would clarify the success criterion where it first appears in §3. “On exact-length tasks, joint success requires a single draft to reach the target word count and contain every required word or phrase. A draft can satisfy both checks and still omit an important fact from the original answer.”

To examine what the verifier can miss, we compared the first passing draft with the draft immediately before it on the same task. On lexical tasks, GPT-6 Luna and Claude Haiku 5.5 more often agreed on the earlier wording. For example, Falcon-H1 7B changed “decided to rename Boetsap” to “decided to renamed Boetsap.” Adding the required word made the sentence pass, but both judges preferred the grammatical earlier version in both presentation orders. This example shows a loss of fluency. Determining whether an edit also preserves the intended meaning requires reading it in the context of the original request. Our Official Comment “Two-judge comparisons of writing quality across drafts” reports the full comparison.

W3. Limited generality

“These tasks allow complete, exact feedback, whereas factual correction and open-ended revision often involve ambiguous or incomplete feedback.”

We chose tasks where a verifier can identify every unmet constraint and give an exact diagnosis at each round. This lets us study how the model acts on feedback while holding feedback quality fixed. It remains open whether the same failure patterns arise in factual correction, where feedback may flag only one of several errors, or in open-ended rewriting, where several edits may plausibly serve the author’s intent. We appreciate your suggestion and would make this scope explicit in the Discussion, with both settings identified as future work.

W4. Empirical contribution

“Its main contribution is therefore a deeper empirical understanding of existing problems rather than a methodological advancement.”

Our contribution is empirical. Systems increasingly rely on models to turn feedback into repairs. We tested that step across 19 model configurations and three constraint families with feasible targets. Even when told exactly how many words to add or remove, some models changed the word count too much, others too little, and some often left it unchanged. These patterns held on unseen texts and helped predict edits in independent revision runs. Across all tasks, model often repeat earlier drafts they generated before. We held the draft and diagnosis fixed and changed only the earlier dialogue. This could help a model escape repetition with little gain in final success. Thank you for helping us clarify why accurate feedback alone can leave a repair unfinished.

Dear authors, thanks for your response and it certainly helps. I will keep my positive rating.

Thank you for reading our response so carefully and for your continued support.

Comments on the paper

4 comments
AuthorsOpenReview ↗

A round-by-round audit of content retention and sentence consistency

We used Stanza to identify named entities and proper names, and rules to identify items such as numbers, URLs, and email addresses. For each category, retention is the percentage of distinct items from the initial draft that still appear in the current draft. We used a DeBERTa-v3-large natural language inference (NLI) model to estimate the probability of contradiction between neighboring sentences within each draft.

How does content change across rounds? We followed the same 480 exact-length tasks per baseline, carrying passing drafts forward. Each column follows the same eligible cases through round 8 and weights the 12 models equally. Values are percentages.

RoundSolved by roundEntity retentionProper-name retentionSymbol retentionAdjacent-sentence contradiction
01.2100.0100.0100.04.84
13.296.098.598.54.80
26.196.298.498.74.69
39.595.698.498.54.70
412.595.998.398.74.69
515.195.598.398.34.72
617.195.898.398.64.67
718.695.498.398.14.75
819.895.898.398.54.70

Success covers 5,760 model–task pairs. Retention is relative to the initial draft (4,515 eligible pairs for entities, 3,545 for names, 1,277 for symbols). The last column averages estimated contradiction probabilities between adjacent sentences within each answer (5,472 pairs scored throughout).

Entity retention fell most on the first edit for every baseline. From round 1 to round 8, success rose from 3.2% to 19.8%, while average retention on all three measures stayed near its round-1 level. Estimated contradiction showed no sustained increase.

What happens to the text when a revision first succeeds? Appendix B.2, Table 11 reports seven additional content checks for two controllers on the exact-length API panel. We can include the round-level results in the revised appendix.

We applied the same frozen annotators to all 12 local baselines on the same 480 exact-length tasks. The Repairs column counts initially failing cases that achieved required constraints finally. We compared the first successful revision with the immediately preceding draft. Entity, name and symbol columns give the mean change in retention of items from the initial draft. NLI gives the change in mean contradiction probability between adjacent sentences within each output. Changes are in percentage points, parentheses give the number of applicable pairs, while add-only loss is reported as n/N: N counts initially too-short drafts containing an entity and all required anchors, n counts those whose first revision lowered entity retention and dropped at least one non-anchor entity name from the text.

ModelRepairsEntityNameSymbolNLIAdd-only loss
Llama 3.1 8B364-0.48 (305)+0.08 (249)0.00 (100)+0.23 (343)1/167
Gemma 2 9B1400.00 (111)-0.10 (92)0.00 (22)+0.13 (134)1/219
GLM-4 9B49-0.28 (33)0.00 (29)0.00 (11)-0.33 (43)8/328
Ministral 8B27+5.26 (19)0.00 (15)0.00 (5)+0.33 (22)2/193
Falcon-H1 7B94+0.36 (69)0.00 (61)0.00 (15)+0.46 (93)25/251
Granite 3.3 8B54-0.29 (42)-2.08 (34)0.00 (12)-1.68 (52)36/169
Qwen3 1.7B60.00 (3)0.00 (1)0.00 (1)+2.84 (6)2/238
Qwen3 4B280.00 (21)0.00 (17)0.00 (5)+0.49 (27)1/246
Qwen3 8B37-0.67 (30)0.00 (22)0.00 (9)+0.13 (36)0/132
Qwen3 14B64+0.27 (53)0.00 (35)0.00 (16)+0.38 (62)0/217
Qwen3 30B-A3B650.00 (45)0.00 (34)0.00 (7)+0.70 (62)2/293
Qwen3 32B140-1.59 (100)-0.70 (83)0.00 (33)-0.73 (127)2/259

Across 12 baselines, we saw no consistent sharp decline in the measured properties at first repair. Mean entity retention rose, held steady or fell by less than one point in 11 models. Qwen3 32B fell by 1.59 points. Mean symbol retention was unchanged in all 12. Adjacent-sentence contradiction probability moved in both directions, by less than one point in 10 models. The largest increase (+2.84 points for Qwen3 1.7B) rests on six repairs.

The clearest model differences appeared in the first edit of a short, entity-bearing draft with all anchors present. Although the feedback asked only for more words, a non-anchor name disappeared and entity retention fell in 21.3% of Granite cases and 10.0% of Falcon cases, versus 0.5% across the six Qwen baselines. We hope this helps clarify where revision carries a measurable content cost. This illustrates another aspect of Exact Feedback Is Not Control: even when feedback specified exactly how many words to add, some edits also removed names already in the draft.

AuthorsOpenReview ↗

Two-judge comparisons of writing quality across drafts

We thank fvyD and rSP2 for asking how revision affects answer quality. Our earlier comment tracked which details survived. GPT-6 Luna and Claude Haiku 5.5 each judged which draft better served the writing task.

The Judges saw anonymous texts in both orders (as Zheng et al. (NeurIPS 2023) reported, LLM judges are sensitive to the order of options they are given). They saw the original request for exact length and assessed clarity, coherence, and completeness of meaning for the sentence or paragraph tasks.

Across 31,682 distinct pairs of nonidentical drafts, swapping the presentation order changed which text Luna chose in 14.6% of pairs and which text Haiku chose in 8.9%. These rates count only switches from choosing one text to choosing the other.

Across 18,399 revised trajectories, we compared each model's initial draft with the first edit and with the final draft on the same task. We count a preference only when both judges pick the same draft in both orders, and the "First-edit margin" and "Final-draft margin" values are positive when the initial draft is preferred more often than the revised draft. For example, if they prefer the initial draft in 30% of all cases and the revision in 10%, the margin is +20 percentage points. Each judge could choose either draft or select TIE if both served the writing task equally well. We counted identical drafts as ties. We included every case when calculating percentages, even when the judges disagreed or no verdict was available.

A joint reversal requires both judges, in both orders, to favor the initial draft at one stage and the revision at the other. Of all 18,399 trajectories, 40 (0.22%) reversed. In 37 the preference shifted toward the initial draft, and in three it shifted away. Among the 2,113 with a joint preference at both stages, 1.9% reversed.

Panel A. API models

Task or modelRevised trajectories (N)First-edit margin (pp)Final-draft margin (pp)Paired change (95% CI)Joint reversal (%)
Exact length3,220+0.6+7.9+7.3 [6.3, 8.3]0.09%
Lexical1,772+16.4+27.1+10.7 [8.4, 13.0]0.62%
Compositional463+8.0+11.4+3.5 [0.4, 6.6]0.00%

On exact length, Luna favored initial over final drafts for Opus 5 (42 vs. 9 cases) and GPT-5.6 Sol (85 vs. 8). Haiku rated both drafts equally good in 82% and 64% of cases, respectively.

Panel B. Locally evaluated models

Task or modelRevised trajectories (N)First-edit margin (pp)Final-draft margin (pp)Paired change (95% CI)Joint reversal (%)
Exact length5,690−0.3+8.2+8.5 [7.4, 9.6]0.19%
↳ Llama 3.1 8B463+5.6+27.4+21.8 [17.5, 26.1]0.22%
↳ Gemma 2 9B478−14.9†−4.0+10.9 [7.1, 14.9]0.63%
↳ GLM-4 9B479−8.8†+8.1+16.9 [12.7, 21.3]0.42%
Lexical4,341+19.8+35.4+15.6 [14.1, 17.0]0.35%
Compositional2,913+5.7+10.1+4.4 [3.4, 5.4]0.00%

Trajectories have equal weight. Paired change is final minus first-edit margin (pp). The 95% intervals use 10,000 task bootstrap samples, keeping all stages and models for each task together.

† Both judges more often preferred the first edit than the initial draft for Gemma (17.8% vs. 2.9%) and GLM (12.5% vs. 3.8%). Two-sided exact binomial tests on the directional counts gave Holm-adjusted p < 0.001 for both cells (171 tests).

In both panels, the margin favoring the initial draft grew most on lexical tasks. On exact length, Llama repaired 364 of 463 drafts, the most among local models. Across those 463 trajectories, the final margin still favored the initial draft by 27.4 points.

Which draft reads better when an edit first meets all the constraints? In 6,229 cases, we compared the first draft that passed every check with the draft just before it. Each percentage is the share of all cases in that row where both judges chose the indicated draft in both orders.

Task or modelCasesJudges agreed on the earlier draftJudges agreed on the passing draft
Exact length3,6383.4%0.5%
Lexical1,30838.8%9.4%
↳ Claude Opus 522016.8%28.6%
Compositional1,28312.0%2.4%

On lexical tasks, the judges more often agreed on the earlier draft. One Falcon-H1 7B edit shows why. To include the required word “renamed,” it changed “decided to rename Boetsap” to “decided to renamed Boetsap.” The sentence now passed, but both judges preferred the original grammar. For Claude Opus 5, agreement more often favored the passing draft. On lexical and compositional tasks, the judges assessed the writing without an independent reference for factual accuracy.

Thank you for suggesting this analysis. We would include these results in the revised manuscript and consider asking human readers to judge a sample of the same pairs.

AuthorsOpenReview ↗

Exact-length post-training and cross-family transfer

Can targeted RL improve revision? We initially tested REINFORCE++ in a 256-task pilot. Following fvyD's suggestion, we extended the run to 512 and 1,024 tasks and evaluated transfer to two untrained constraint families.

Training design

We sampled exact-length tasks equally from Dolly, No Robots, OASST1 and WritingPrompts. For the two COLLIE-derived families, we had too few independent items to build comparable training sets, so they served only as test sets. We selected the reward and hyperparameters through repeated small-sample experiments on Qwen3-1.7B. From a stored draft, Qwen3-8B sampled up to eight revisions using the full history and exact verifier feedback. An exact count, truncation or the turn limit ended the rollout. Only revision tokens entered the update. Each task was visited once, four per update. The 256-, 512- and 1,024-task checkpoints are steps 64, 128 and 256 of one seeded run.

Let dtd_t be absolute word-count error after edit tt, with t=0t=0 the given draft. Set Pt=(1+dt)−2P_t=(1+d_t)^{-2} and M=0.9TPTM=0.9^T P_T after TT edits (M=0M=0 for a truncated revision). Let RR count edits repeating a previously seen normalized word sequence and SS count other edits that failed to reduce the error. The reward for edit tt is

rt=Pt−Pt−1+1t=T[M(1−0.005R−0.001S)−PT].r_t=P_t-P_{t-1}+\mathbf{1}_{t=T}\big[M(1-0.005R-0.001S)-P_T\big].

Moving closer earns immediate credit. Earlier exact repairs earn more.

REINFORCE++ update: each edit's last token receives its reward. Let GiG_i be token ii's task reward-to-go, standardized as z(Gi)z(G_i) across revision tokens in the batch. For the sampled token log-ratio kj=log⁡(πold/πref)k_j=\log(\pi_{\rm old}/\pi_{\rm ref}), set Bi=z(Gi)−0.01∑j≥ikjB_i=z(G_i)-0.01\sum_{j\ge i}k_j within the rollout. Our adjustment sets Ai=min⁡(Bi,−2)A_i=\min(B_i,-2) on tokens of a repeated, unsuccessful edit that failed to reduce the error, and Ai=BiA_i=B_i elsewhere. With sampled token ratio ui=πθ/πoldu_i=\pi_\theta/\pi_{\rm old}, we maximize

J(θ)=mean⁡imin⁡(uiAi,clip⁡(ui,0.8,1.2)Ai).J(\theta)=\operatorname{mean}_i\min\big(u_i A_i,\operatorname{clip}(u_i,0.8,1.2)A_i\big).

The cap prevents a later repair from giving positive credit to a failed repeat. We used FP32 LoRA on seven attention and MLP projections (rank 32, alpha 64, AdamW learning rate 5e-6).

Each checkpoint generated its own drafts, with up to eight greedy revisions per case. Exact-length success requires the target count and all anchors. Neither transfer panel matched training items on IDs, normalized instructions or reference text.

Final joint success (% of 480 cases per family).

Training tasksExact lengthLexicalCompositional
Paper baseline9.6%35.4%69.4%
25620.0%36.0%72.3%
51226.0%39.0%77.3%
1,02426.9%38.1%80.2%

Early recurrence (% of active revisions in rounds 1–4 that failed and exactly repeated an earlier output).

Training tasksExact lengthLexicalCompositional
Paper baseline66.8%49.1%77.2%
25649.4%43.1%66.7%
51239.4%38.1%54.3%
1,02430.4%31.9%46.2%

What transferred? Exact-length training improved final success over baseline in all three families. Early recurrence fell at every checkpoint, including on the untrained families. Later training still improved compositional success, while exact-length success nearly plateaued and lexical success dipped. Avoiding a repeated draft transfers more consistently than completing a repair.

Writing quality after RL. GPT-6 Luna and Claude Haiku 5.5 compared the baseline and 1,024-task checkpoints’ anonymous final answers on the same 480 tasks per family. A preference required both judges to choose the same answer in both orders. Percentages use all 480 pairs, including identical answers and ties.

TaskBoth preferred baselineBoth preferred RL
Exact length66 (13.8%)24 (5.0%)
Lexical81 (16.9%)36 (7.5%)
Compositional23 (4.8%)8 (1.7%)

Formal success rose in every family, yet the judges more often agreed on the baseline answer. On compositional tasks, they agreed on a tie in 299 pairs. They saw the original request for exact length and assessed clarity, coherence, and completeness of meaning on the transfer tasks.

Training cost. Training took 14.4 GPU-hours, excluding earlier reward tuning on Qwen3-1.7B. The final 512 tasks took 6.5 GPU-hours and yielded four additional exact-length successes, while compositional success continued to improve.

Where might revision ability come from? In Appendix E (Table 22), the best-performing Tülu stage depends on the constraint family. We hypothesize that correcting derivations, code, or prose can teach models to change what feedback identifies while preserving what already works. The checkpoint comparison cannot identify which training tasks taught this skill.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Revisions (for AC, SAC, and Reviewers)

Dear Reviewers, Area Chair and Senior Area Chair,

We sincerely thank you and Reviewers fvyD, HtwS, and rSP2 for your constructive feedback—especially rSP2 for maintaining a positive rating and HtwS for raising their score. In Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs, we study closed-loop revision under exact deterministic feedback across 19 models (12 local, 7 API), three constraint families, and eight rounds (27,360 trajectories). Separating error observability from revision control, we show that exact diagnostics alone do not ensure completion due to reproducible editing biases and dialogue-history dynamics. Addressing questions on answer quality, RL transfer, novelty, and clarity, we summarize our additions below:

  1. Content preservation and two-judge writing quality evaluation. To test whether satisfying formal constraints degrades answer quality, we audited 5,760 local exact-length trajectories using Stanza and DeBERTa-v3-large NLI. Initial-draft entity retention remained high through round 8 with no sustained rise in adjacent-sentence contradiction, though models such as Granite 3.3 8B and Falcon-H1 7B still dropped non-anchor entity names when asked only to add words. Complementarily, dual-judge evaluations (GPT-6 Luna and Claude Haiku 5.5, scored in both presentation orders with added constraints hidden) across 18,399 revised trajectories revealed a clear constraint–quality trade-off: final drafts were preferred less often than initial drafts, and at the first edit passing all lexical checks, both judges favored the preceding draft far more often than the passing draft (38.8% vs. 9.4%, often due to sacrificed grammar and fluency), whereas Claude Opus 5 reversed this trend (fvyD-W1&Q1; rSP2-W2).

  2. Targeted RL post-training and cross-family transfer. Following fvyD's suggestion, we trained Qwen3-8B with REINFORCE++ on exact-length revision up to 1,024 tasks (14.4 GPU-hours) and evaluated zero-shot transfer to the two untrained constraint families. Final success improved on exact-length (9.6% to 26.9%) and transferred to both untrained families (most notably compositional, 69.4% to 80.2%), while early output repetition dropped substantially across all three families—showing that avoiding repeated drafts transfers more consistently than completing repairs. However, both judges still preferred the baseline's final answers over the RL checkpoint's across every task family, confirming that optimizing directly for exact feedback does not automatically preserve writing quality (fvyD-W2&Q2).

  3. Conceptual novelty, failure mechanisms, and task scope. We clarified how holding target feasibility and verifier completeness fixed isolates model-side feedback-to-action responses that prior benchmarks (DeCRIM, Stechly et al., RefineBench, Meeseeks) could not disentangle. Specifically, our framework uncovers (i) reproducible, checkpoint-specific edit patterns (under-correction, directional asymmetry, and near-zero action) that generalize to unseen texts and do not monotonically improve with model scale or post-training stages (§4.2, App. E), and (ii) widespread verbatim output repetition across failed trajectories, where removing retained dialogue at matched recurrent states can substantially shift next-step escape (helping or hindering depending on the model and task) while barely moving final success (§5.1–§5.2). We also clarified why non-LLM baselines (constrained decoding, deterministic editing) do not test whether an LLM can act on exact feedback (HtwS-W1, W2, Q1, Q2; rSP2-W4).

  4. Manuscript clarity, definitions, and claim refinement. We prepared concrete revisions across the Abstract, Contributions, §3–§5, and Discussion to split dense sentences and define technical terms (including next-step escape and zero-action) before reporting results. We also explicitly distinguished formal constraint satisfaction from preserving response meaning and quality, and framed open-ended rewriting and factual correction under incomplete feedback as future work (rSP2-W1, W2, W3; HtwS-W2).

We believe these clarifications and additional analyses substantially strengthen the soundness, clarity, and empirical scope of our work. All corresponding additions will be integrated into the revised manuscript, and detailed analyses are provided in our official comments and point-by-point responses. In particular, we hope Reviewer fvyD may find a moment in their busy schedule during the upcoming discussion phase to review our supplementary experiments and analyses, and we would greatly appreciate their thoughts on whether these additions address their questions.

Authors of Submission 18037

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.