acceptodds
Under review as a conference paper at ICLR 2027

Solve, Recover, Reuse: LLM Reasoning Across Workflows with Corrupted Contexts

Abstract

Large language models (LLMs) increasingly operate in complex workflows and noisy contexts. It's important to have a comprehensive understanding of how model performance varies across workflow and corruption types, and whether both harmful and beneficial transitions arise under these challenges. We introduce an Evolutionary Task Generator (ETG) that generates a series of linked tasks across workflows and corruptions, providing a framework to systematically experiment and analyze: 1) model performance changes across workflows and corruptions, 2) the bidirectional transitions under corrupted contexts, and 3) training effects across workflows and corruptions. Across frontier LLMs, we find that model rankings and performance change substantially across workflows, e.g., DeepSeek-V4-Flash drops from 0.75 in direct solving to 0.20 in downstream reuse. Instance-level analysis shows that both harmful and beneficial transitions arise under corrupted contexts. Training results show that direct-solving gains may not transfer across workflows, and may even degrade model performance under complex workflows with corrupted contexts. Our results highlight the importance of examining LLMs reasoning through the lens of complex workflows and corrupted contexts, and suggest a more constructive direction when such noise is unavoidable in practice: resist the harmful and leverage the beneficial.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning, Robustness, Corrupted Context

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_JhctOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Evolutionary Task Generator (ETG), a lineage-preserving and graph-annotated framework for studying LLM reasoning across three workflow modes: direct solving, explicit recovery from a corrupted interaction, and downstream reuse of the recovered state. Starting from a common synthetic reasoning graph, ETG constructs six corruption types by crossing two corruption targets (value and dependency) with three interaction regimes (static corruption, unsupported assumption, and evidence-integration failure). The paper further defines two instance-level transition metrics: αi\alpha_i, measuring failure-to-success transitions between consecutive modes, and βi\beta_i, measuring success-to-failure transitions.

Critical strengths
  1. Constructing Solve, Recover, and Reuse tasks from the same seed problems enables instance-level comparisons across workflows and corruptions, revealing that strong direct-solving performance does not guarantee reliable recovery or downstream performance under corrupted contexts. This provides a more informative evaluation of LLM reasoning.
  2. The experiments show that clean-only fine-tuning can improve direct-solving accuracy while degrading recovery and downstream performance, whereas mixed training generally yields more balanced improvements. These findings highlight the importance of evaluating training strategies across workflows rather than relying solely on direct-solving gains.
Critical weaknesses
  1. Although the paper explicitly treats failure-to-success transitions as descriptive rather than causal, these transitions do not establish whether models actually leverage useful information from the corrupted history or simply ignore it and solve the task again. The ‌repeated‌ claim of "resist the harmful and leverage the beneficial" lacks direct experimental support.
  2. Mode 3 only evaluates corrupted histories, without comparing against correct histories or no prior history on the same downstream tasks. This makes it difficult to determine whether models can selectively identify reliable information or simply distrust prior answers.
Questions for authors
  1. In Table 2,Claude-Opus-5 achieves 0.98 accuracy on 200 Mode 1 seed tasks,leaving only four initially failed seeds.Since α₁ is derived from these four seed-task lineages,how statistically reliable is the reported high α₁?
  2. The paper shows performance differences across workflow modes and corruption types,but the main contributing factors remain unclear.For example,why are unsupported assumptions harder to recover from?What are the relative impacts of corruption target, interaction regime, turns or roles?Could the authors clarify these factors and provide insights for future model evaluation and training?
  3. Mode 3 constructs downstream tasks using a simple linear relation,y₂=f(y₁).Would the paper's main findings still hold for downstream tasks involving more complex,multi-step dependencies?
  4. Could the authors report the counts and proportions of all six corruption types in mode?Are the overall results driven by particular types?Please also provide a few paired failure-to-success and success-to-failure examples to show what changes in the model’s responses.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Jhct (1/4)

We greatly appreciate your recognition of our informative reasoning evaluation and the practical implications of our training findings. Thank you for your thoughtful suggestions on strengthening the analysis and experimental evidence. We address your concerns below.


W1: Further Evidence for Transitions

Mode 1 provides a no-context reference for the idealized case of “simply ignoring the corrupted context.”

Specifically, α₁ measures the proportion of Mode-2 instances answered correctly among those evolved from seed tasks that the model answered incorrectly in Mode 1. These transitions therefore capture cases where the model fails without context but succeeds with corrupted context. Table 4 reports α and β; for example, GPT-5.5 achieves α₁ 0.444.

We also selected the instances models answered correctly in Mode 3 and evaluated those same instances without context. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.

ModelMode-3 AccuracyNo-Context Mode-3 Accuracy
GPT-4o1.0000.788
GPT-5.41.0000.836

The no-context condition serves as a proxy for ignoring the context. Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information in the added contexts.

Regarding training, Figure 4 shows that mixed training increases α and decreases β. Here, α captures beneficial failure-to-success transitions, where the model answers incorrectly in the preceding mode but correctly in the current mode. Conversely, β captures harmful success-to-failure transitions, where the model answers correctly in the preceding mode but incorrectly in the current mode.

Building on these findings, the phrase “suggest a more constructive direction” in our abstract (Line 26) is intended to introduce a future research direction:

suggest a more constructive direction when such noise is unavoidable in practice: resist the harmful and leverage the beneficial.

AuthorsOpenReview ↗

Response to Reviewer Jhct (2/4)

W2: Mode-3 Ablation Study

We thank the reviewer for suggesting insightful experiments on Mode 3.

We selected the instances models answered correctly in Mode 3 and evaluated those same instances without context. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.

ModelMode-3 AccuracyNo-Context Mode-3 Accuracy
GPT-4o1.0000.788
GPT-5.41.0000.836

The no-context condition serves as a proxy for ignoring the context. Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information in the added contexts.

Below is a concrete example from DeepSeek’s output in which the model explicitly corrects the upstream calculation and uses the corrected result to answer the downstream question, rather than simply ignoring the context:

Correcting the earlier calculation: Fashion Backpack has 6 Text Markers and the same number of Highlighters, so 6 Highlighters.

Total stationeries: 6+6=126 + 6 = 12.

Dollars: 12+4=1612 + 4 = 16.

The final answer is <<16>>.

AuthorsOpenReview ↗

Response to Reviewer Jhct (3/4)

Q1

Claude's very high accuracy in Mode 1 left relatively few failure cases available for transition analysis. To expand this analysis, we identified 24 tasks that Claude answered incorrectly among 1,300 Mode-1 tasks and generated 795 Mode-2 instances from these seed tasks.

Our conclusion remains unchanged: beneficial transitions are non-negligible, with α₁ = 0.634 (95% CI: [0.523, 0.738]).

For α₁, we also report the estimates and 95% CIs for the other models, together with the numbers of Mode-1 seed tasks and corresponding evolved Mode-2 instances.

GPT-5.5GPT-5.4GPT-4oGemini-3.6-FlashDeepSeek-v4-Flash
α₁ (95% CI)0.444 [0.359, 0.533]0.194 [0.151, 0.240]0.232 [0.208, 0.257]0.339 [0.248, 0.430]0.343 [0.280, 0.409]
Seeds / Evolved Instances22 / 33382 / 1223102 / 149636 / 54950 / 747

Q2

The role ablation provides further insight into cases of unsupported assumptions. (Line 303) When we change the role introducing the unsupported assumption from Assistant to User, accuracy improves substantially. In some settings, accuracy becomes comparable to that in other Mode-2 settings. As discussed in Line 303, the lower performance in the User-Assistant configuration suggests stronger anchoring on the assistant’s false beliefs.

We also report average accuracy grouped by several factors below.

FactorCategoryMode-2 Acc.
Corruption TargetValue (Node)0.6015
Corruption TargetDependency (Edge)0.5997
Interaction RegimeStatic Corruption0.7499
Interaction RegimeUnsupported Assumption0.4032
Interaction RegimeEvidence-Integration Failure0.6491
TurnsSingle-turn0.7499
TurnsMulti-turn0.4920

Among these grouped averages, interaction regime exhibits the largest performance gap, with Unsupported Assumption yielding the lowest accuracy. The difference between corruption targets is relatively small. Multi-turn accuracy is substantially lower than single-turn accuracy.

These findings highlight the importance of extending model evaluation and training beyond idealized clean contexts to include corrupted contexts, particularly unsupported assumptions introduced by the models themselves in multi-turn dialogues.


Q3

We extended our Mode-3 evaluation to two-step downstream tasks, and our main findings continue to hold.

ModelAcc.Misled Rateα₂β₂
GPT-4o (1-step)0.06360.97840.03880.8711
GPT-4o (2-step)0.05350.99360.03310.8928
GPT-5.4 (1-step)0.17330.95200.07440.7304
GPT-5.4 (2-step)0.08240.98860.02960.8663

Models remain susceptible to being misled. Compared with one-step tasks, two-step tasks yield lower accuracy and a higher misled rate among parseable errors.

Reviewer_tsp4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors investigated how LLMs reason across linked multi-step workflows when operating over structurally corrupted interaction histories.. The proposed Evolutionary Task Generator (ETG) has 3 operational modes: Direct Solving (Mode 1), Explicit Recovery (Mode 2), and Downstream Reuse (Mode 3). Authors evaluated 6 LLMs and conducted fine-tuning experiments on smaller open-weight models. The study established that model rankings shift sharply across workflows. Additionally, it tracks instance-level bidirectional transitions and shows that mixing clean and corrupted-recovery training trajectories improves cross-workflow generalization to unseen downstream reuse tasks.

Critical strengths
  1. Authors demonstrate that evolving the exact same problem across Direct Solving, Explicit Recovery, and Downstream Reuse reveals that models with 97%–100% explicit error detection in Mode 2 still blindly reuse corrupted answers 75%–99% of the time when failing follow-up questions in Mode 3, reversing single-turn leaderboard rankings.
  2. Authors present valuable Post-Training & Role-Anchoring Insights. For example, replacing user–assistant labels with user1–user2 exposes strong self-anchoring on prior assistant mistakes. Additionally, fine-tuning solely on clean Mode-1 problems degrades multi-turn recovery, whereas a 1:1 mix of clean and corrupted-recovery data improves all workflows and transfers zero-shot to unseen Mode 3.
  3. Authors show that mixing clean and error-recovery training data generalizes zero-shot to downstream reuse.
Critical weaknesses
  1. Mode 1 is zero-shot, whereas Modes 2 and 3 include a worked assistant response demonstrating modulo-23 rules and answer formatting, explaining why untuned Qwen3-4B scores nearly four times higher on corrupted Mode-2 Static (47.0%) than clean Mode 1 (12.5%) and inflating failure-to-success transitions.)

  2. For top models , Mode-1-to-Mode-2 failure-to-success rates are computed from just 4 failed seed problems; high Mode-2-to-Mode-3 failure-to-success rates largely reflect models rebounding to their high Mode-1 competence (more than net gains over clean solving) Mode 2 has no clean negative controls to test false-positive error detection.

  3. There are missing prior-art baselines and the scope is narrow and synthetic. For example all reported experiments are done to synthetic iGSM modulo-23 math without benchmarking against inline mistake-correction training on the same dataset or testing out-of-domain generalization.

Questions for authors
  1. What are the exact user prompt templates for Modes 1, 2, and 3, and what parsing rule extracts binary error detection separately from the final answer in Mode 2?
  2. How much do Mode 1accuracy and the Mode 1 to Mode 2 failure to success rates change if Mode 1 includes a single clean example in the prompt?
  3. How are transition rates weighted across seed lineages of different depths?
  4. Can you report Mode-3 accuracy in Table 2 broken down by regime (Static, Assumption, Integration) along with direct Mode 1 to Mode-3 transition rates?
  5. How does your multi-turn recovery training compare against single-stream mistake-correction training on iGSM, and do your Mixed SFT models improve on any non-synthetic tasks without modulo-23 arithmetic?
6 comments
AuthorsOpenReview ↗

Response to Reviewer tsp4 (1/4)

We greatly appreciate your recognition of the diagnostic value of our evaluation and the practical relevance of our findings for model training and generalization. Thank you for your detailed suggestions on experimental controls, clarity, and comparisons. We address your concerns below.


W1: One-Shot Ablation

We agree with the reviewer that differences between zero-shot and few-shot prompting could affect the results. To verify existing conclusions, we conducted an additional experiment by adding a single clean example to the prompt, making Mode-1 from zero-shot to one-shot.

The table below compares the original setup with this one-shot ablation.

ModelMode-1 SettingMode-1 Acc.α₁β₁
GPT-4oOriginal (Zero-shot)0.490.2320.6761
GPT-4oOne-shot0.560.22630.6823
Qwen3-4BOriginal (Zero-shot)0.1250.25980.4127
Qwen3-4BOne-shot0.1250.26120.3966

Our conclusion remains unchanged: both beneficial failure-to-success transitions (α₁) and harmful success-to-failure transitions (β₁) remain non-negligible.


W2: Reliability of Transition Studies

Claude’s very high accuracy in Mode 1 left relatively few failure cases available for transition analysis. To expand this analysis, we identified 24 tasks that Claude answered incorrectly among 1,300 Mode-1 tasks and generated 795 Mode-2 instances from these 24 seed tasks. Our conclusion remains unchanged: beneficial transitions are non-negligible, with α₁ = 0.634 (95% CI: [0.523, 0.738]).

Furthermore, we selected the instances models answered correctly in Mode 3 and evaluated them under a no-context condition. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.

ModelMode-3 Acc.No-Context Mode-3 Acc.
GPT-4o1.0000.788
GPT-5.41.0000.836

Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information in the added contexts.

We also clarify our observation (Line 323): although beneficial transitions occur in Mode 3, α₂ may not be as high as α₁. Downstream reuse is more difficult than recovery.

We agree that our original detection experiments evaluated only corrupted contexts. We therefore added a clean-context detection experiment for Mode 2. GPT-4o achieved detection accuracies of 66.83%. The result shows that the models can recognize clean contexts, rather than invariably treating all inputs as erroneous.

Please refer to Q4 for direct Mode-1 to Mode-3 transition rates. Our conclusion remains unchanged: both beneficial failure-to-success transitions and harmful success-to-failure transitions remain non-negligible.

AuthorsOpenReview ↗

Response to Reviewer tsp4 (2/4)

W3: Broader Scope Experiments

We added an inline mistake-correction training baseline, in which an error in the output is immediately followed by a self-correction. This baseline and our approach investigate different aspects of training: inline mistake-correction focuses on where corrections are inserted, whereas our approach studies the mixing ratio of clean and corrupted training data.

We also evaluated their combination (Inline-Mixed). Combining the two approaches improves Inline performance in Mode-1 and Mode-2 settings.

TrainingMode-1 Acc.Mode-2 Static Acc.Mode-2 Assumption Acc.Mode-2 Integration Acc.Mode-3 Acc.
Mixed0.60.71370.72330.47330.4373
Inline0.6250.23080.01940.10690.4815
Inline-Mixed0.650.29060.19420.11450.3838

We also analyze an out-of-distribution (OOD) setting in the manuscript (Line 419):

As our training data contain only Mode-1 and Mode-2 examples, Mode-3 provides a direct test of cross-workflow generalization. Table 5 shows that training with corrupted data improves Mode-3 accuracy while reducing the misled rate, with both changes exceeding 10%.

Furthermore, we evaluated the generalization of mixed training to a new task type: kinship reasoning using CLUTRR ( A Diagnostic Benchmark for Inductive Reasoning). On this task, denoising accuracy increased from 0.05 for the no-training baseline to 0.266 after mixed training.

AuthorsOpenReview ↗

Response to Reviewer tsp4 (3/4)

Q1: Prompts

All three modes share the same system prompt (Line 613).

Mode 2: Explicit Recovery

Mode 2 uses a detection call followed by a separate recovery call only when the detection response is parsed as 1.

The detection user message is:

Here is a case containing an original problem and a candidate solution.
Check whether the candidate solution has any obvious error with respect to the original problem.
If there is an error, return 1. If there is no error, return 0 and we will enter the next step.
Return exactly one character: 0 or 1. Do not explain your decision.

If the detection response is parsed as 1, it is retained as an assistant message, followed by this recovery user message:

You identified an error. Find the first error relative to the original problem, then correct the solution and answer.
Use exactly these three fields:
<brief error type>
<corrected reasoning or corrected solution>
The final answer is <<x>>.
The last line must use that exact format, and x must be an integer from 0 to 22.

Detection decisions and final answers are parsed separately from their respective responses. The recovery-answer parser first looks for a final line matching the format The final answer is <<x>>.

Mode 3: Downstream Reuse

The downstream-reuse user message is:

I have one more question. The number of dollars I have is [OFFSET] [more/less] than the number of [UPSTREAM QUANTITY]. How many dollars do I have?
Reviewer_9UFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an Evolutionary Task Generator (ETG) to synthesize a series of linked tasks for evaluating changes in model performance under corrupted contexts. It aims to show that both improvements and degradations can occur.

Critical strengths

I find the data synthesis approach reasonable: starting from the same seed task, representing the reasoning as a tree, and progressively introducing perturbations is a useful way to generate a series of related tasks.

Critical weaknesses
  1. The writing is difficult to follow. I find the paper somewhat roughly written, possibly with substantial LLM involvement. For example, in lines 503–504, the authors even appear to have included their prompt, “not be more than 1 page.”, in the paper. This raises serious concerns for me about the quality of the paper.

  2. The experiments do not seem sufficiently solid. The paper’s main contribution is to synthesize some math data and report changes in model performance on these data. My question is whether these changes are more general: do they also hold across a broader range of domains and established benchmarks, and do they help the community better understand the mechanisms underlying LLMs? I think some conclusions may be too strong, for example: “As LLMs move from idealized problem solving to complex workflows, improving direct-solving ability alone is insufficient and may even degrade performance in other workflows.”

  3. Have the authors performed quality checks on the synthetic data? Why should we consider the generated question–answer pairs sufficiently high-quality to evaluate state-of-the-art LLMs such as GPT-5.5 and Claude Opus 5.5?

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9UFQ (1/2)

We appreciate your recognition of the methodological value of our task-generation approach, as well as your thoughtful feedback on presentation, data quality, and generalizability. We address your concerns below.


W1: Typographical Error

Thanks for the remind. We will correct this typo and conduct a complete check in the camera-ready version. Due to our unfamiliarity with this new required section of AI USE STATEMENT, we inadvertently left part of the template instruction (“This statement should not be more than 1 page.”) in the manuscript. This is a type of error that humans might make, but LLMs may not. Thanks again for the remind.


W2: Experiments and Conclusions

We used “may” to express a possibility rather than a universal claim. We agree that the original statement would benefit from more precise qualification. We therefore clarify our intended conclusion as follows:

As LLMs move from idealized problem solving to complex workflows, improving direct-solving ability alone, under certain settings, is insufficient and may degrade performance in other workflows.

This conclusion is supported by the following observations. First, clean training sometimes yields lower accuracy in Modes 2 and 3 than the no-training baseline (Line 442). Second, clean training sometimes increases β, indicating a higher rate of success-to-failure transitions (Figure 4).

These findings highlight the importance of examining LLM reasoning through the lens of complex workflows and corrupted contexts. Models can be misled when performing downstream tasks, while corrupted contexts can have both beneficial and harmful effects, reflected in failure-to-success transitions (α) and success-to-failure transitions (β), respectively.

We further evaluated whether these conclusions extend to a different task type: kinship reasoning, using CLUTRR (A Diagnostic Benchmark for Inductive Reasoning). We evaluated 3640 instances in total: 280 in Mode 1, 1680 in Mode 2, and 1680 in Mode 3.

ModelDirect Acc.Denoise Acc.Reuse Acc.α₁β₁α₂β₂
GPT-4o0.77500.66900.25360.42060.25880.09170.6664
GPT-5.40.88930.86850.77080.43010.07700.47510.1844

These additional results support our conclusions that model performance can be influenced by work modes and that corrupted contexts can have both beneficial and harmful effects.

AuthorsOpenReview ↗

Response to Reviewer 9UFQ (2/2)

W3: Quality Checks

We generate tasks programmatically and validate their correctness using an additional official verification tool. Specifically, we first generate the graph structure of each task and then derive natural-language question–answer pairs from it. An additional validation module performs correctness checks for each variable. Furthermore, before training and evaluation, we sampled instances from every mode and task type for manual inspection. All sampled instances passed these checks.

This procedure follows the same underlying quality-control pipeline used in prior work on GSM tasks [1].

[1] Ye, Tian, et al. “Physics of language models: Part 2.1, grade-school math and the hidden reasoning process.” International Conference on Learning Representations (ICLR), 2025.

Reviewer_yJrWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that solving a problem, explicitly repairing faulty reasoning, and correctly reusing information from a faulty history are distinct capabilities—and that direct-solving benchmarks alone miss important reliability failures.

Contributions

  • Evolutionary Task Generator (ETG): A scalable, graph-annotated generator that derives linked Solve, Recover, and Reuse tasks from the same seed, enabling instance-level comparisons.
  • Controlled corruption taxonomy: Six corruption types crossing value vs. dependency errors with static errors, unsupported assumptions, and evidence-integration failures.
  • Bidirectional analysis: Measures failure→success transitions (α) and success→failure transitions (β), rather than reporting only aggregate degradation. These are descriptive workflow transitions—not evidence that corruption itself causes improvement.

Findings and validation

  • Frontier-model evaluation: Six models tested on 5,636 tasks derived from 200 seeds, spanning 1–8 reasoning steps in synthetic modulo-23 arithmetic. Performance and rankings change substantially: Claude scores 98% on Solve but 58.2% on Reuse, whereas GPT-5.5 scores 89% and 73.2%, respectively. Many downstream failures reproduce the corrupted upstream answer.
  • Controlled training: Two small models receive clean-only, corrupted-recovery-only, or mixed supervision, with 30,000 examples per run. Clean-only training can worsen recovery and reuse; mixed training generally provides more balanced gains. Reuse is excluded from training, making its improvement evidence of cross-workflow transfer.
  • Robustness checks: Role-label and wording ablations, 95% seed-cluster bootstrap confidence intervals, and repeated evaluations support the principal patterns. Appendix DPO/GRPO and autonomous-data-generation experiments provide exploratory evidence.
Critical strengths
  • A controlled framework for comparing reasoning across workflows. ETG derives Solve, Recover, and Reuse evaluations from the same seed problems, with explicit reasoning graphs and systematically constructed errors. This enables instance-level comparisons that aggregate benchmark scores cannot provide. The seed-level bootstrap analysis appropriately accounts for dependence among derived tasks, strengthening confidence in the reported workflow differences.
  • Consequential evidence that solving ability and context-handling reliability diverge. Model rankings change across workflows, and clean-only fine-tuning can improve solving while degrading recovery and reuse. Conversely, mixed solving-and-recovery training improves downstream reuse despite including no reuse examples—for example, Qwen’s reuse accuracy rises from 25.5% to 43.7%. These findings make the framework practically informative: they show why model selection and training should evaluate context recovery and reuse separately rather than assume improvements transfer automatically.
Critical weaknesses
  • The evaluation only narrowly represents complex workflows. All tasks are synthetic modulo-23 arithmetic problems, and downstream reuse adds a simple linear transformation to a supplied corrupted history. It does not test whether a model actually preserves and reuses its own repaired state through an evolving interaction. This limits the evidence for the paper’s broader motivation around practical workflow reliability; another domain or an executed recovery-to-reuse pipeline would materially strengthen that connection.
  • The distinctive “beneficial transitions” finding is insufficiently resolved. Failure→success transitions compare different prompts and, for Reuse, different targets, without matched clean-context controls to distinguish useful reasoning scaffolding from other workflow changes. Some estimates also rest on very few independent failures—Claude’s 98% Solve accuracy leaves only four failed seeds—and transition-specific uncertainty is not reported. The paper correctly acknowledges that these transitions are descriptive, but the evidence remains insufficient to establish a robust explanation of when and why imperfect context helps, weakening one of its central contributions.
Questions for authors
  • How well-supported are the failure→success transition estimates? Please report the number of independent seed problems contributing to each α estimate and seed-clustered confidence intervals, particularly for Claude, whose Solve failures comprise only four seeds. This would determine whether the highlighted beneficial transitions are robust or dominated by a few task lineages.

  • Were the training comparisons repeated across random seeds, and were the reported checkpoints selected using the same rule? If repeated results are available, please provide the variability of the clean-only degradation and mixed-training reuse gains. These findings drive the paper's training recommendations, so their stability would materially affect my confidence.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yJrW (1/2)

We greatly appreciate your recognition of our methodological contribution, analytical rigor, and practical significance for model evaluation and training. Thank you for your thoughtful suggestions on strengthening the scope and robustness of our evidence. We address your concerns below.


W1: Additional Experiments on Workflows

We thank the reviewer for the insightful suggestion to evaluate an executed recovery-to-reuse pipeline, which has broadened our perspective on workflow evaluation.

We selected three categories of histories from our existing Mode-2 results: (1) the error was not detected, (2) the error was detected and correctly repaired, and (3) the error was detected but incorrectly repaired. We then evaluated the recovery-to-reuse pipeline using these histories and report the accuracy at the reuse stage (Mode 3) below, alongside the original Mode-3 accuracy.

ModelMode-2 OutcomeOriginal Mode-3 AccuracyRecovery-to-Reuse Accuracy
GPT-5.4Error not detected0.250000.09375
GPT-5.4Error detected and correctly repaired0.312501.00000
GPT-5.4Error detected but incorrectly repaired0.031250.00000
GPT-4oError not detected0.031250.06250
GPT-4oError detected and correctly repaired0.093750.93750
GPT-4oError detected but incorrectly repaired0.093750.09375

The results show that successful recovery substantially improves downstream reuse accuracy. When recovery fails, recovery-to-reuse accuracy can remain low and, in some cases, fall below the original Mode-3 accuracy.

This additional pipeline evaluation complements the original purpose of Mode 3: to test whether a model can recognize upstream errors without being explicitly prompted to look for them when the focus shifts to a downstream task.

Furthermore, we evaluated a new task type, kinship reasoning, using CLUTRR (A Diagnostic Benchmark for Inductive Reasoning). We evaluated 3640 instances in total: 280 in Mode 1, 1680 in Mode 2, and 1680 in Mode 3.

ModelMode-1 Acc.Mode-2 Acc.Mode-3 Acc.α₁β₁α₂β₂
GPT-4o0.77500.66900.25360.42060.25880.09170.6664
GPT-5.40.88930.86850.77080.43010.07700.47510.1844

These results provide further support for our conclusions: model performance can be influenced by work modes. Corrupted contexts can have both beneficial and harmful effects, as reflected in the α and β metrics.


W2: Reliability of Transition Studies

Our task evolution method uses controlled comparisons between modes. As shown in Fig. 1(a) and (b), Mode 1 and Mode 2 use the same seed task and share the same conditions H; Mode 2 only adds the corrupted dialogue. Similarly, Mode 2 and Mode 3 share the same seed task and corrupted dialogue; Mode 3 only adds the query for y₂.

Furthermore, for each model, we selected the instances it answered correctly in Mode 3 and evaluated those same instances under a no-context condition. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.

ModelMode-3 Acc.No-Context Mode-3 Acc.
GPT-4o1.0000.788
GPT-5.41.0000.836

The no-context condition serves as a proxy for ignoring the context. Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information contained in the corrupted contexts.

Claude's very high accuracy in Mode 1 left relatively few failure cases available for transition analysis. To expand this analysis, we identified 24 tasks that Claude answered incorrectly among 1,300 Mode-1 tasks and generated 795 Mode-2 instances from these seed tasks.

Our conclusion remains unchanged: beneficial transitions are non-negligible, with α₁ = 0.634 (95% CI: [0.523, 0.738]).

For α₁, we also report the estimates and 95% CIs for the other models, together with the numbers of Mode-1 seed tasks and corresponding evolved Mode-2 instances.

GPT-5.5GPT-5.4GPT-4oGemini-3.6-FlashDeepSeek-v4-Flash
α₁ (95% CI)0.444 [0.359, 0.533]0.194 [0.151, 0.240]0.232 [0.208, 0.257]0.339 [0.248, 0.430]0.343 [0.280, 0.409]
Seeds / Evolved Instances22 / 33382 / 1223102 / 149636 / 54950 / 747
AuthorsOpenReview ↗

Response to Reviewer yJrW (2/2)

Q1

For the 95% CIs and sample counts for α₁, please refer to our response to W2. The corresponding results for β₁, α₂, and β₂ across all six models are reported below. “Seeds / Evolved Instances” denotes the number of seed tasks from the preceding mode and the number of evolved instances in the current mode.

MetricGPT-5.5GPT-5.4GPT-4oGemini-3.6-FlashDeepSeek-v4-FlashClaude-Opus-5
β₁ (95% CI)0.255 [0.235, 0.275]0.247 [0.213, 0.282]0.676 [0.644, 0.706]0.206 [0.185, 0.228]0.297 [0.270, 0.326]0.189 [0.168, 0.209]
β₁ Seeds / Evolved Instances177 / 2432117 / 154297 / 1269163 / 2216149 / 2018195 / 2703
α₂ (95% CI)0.652 [0.601, 0.705]0.074 [0.056, 0.095]0.039 [0.023, 0.056]0.445 [0.387, 0.511]0.138 [0.116, 0.163]0.427 [0.377, 0.477]
α₂ Seeds / Evolved Instances179 / 775174 / 1317198 / 1934165 / 793184 / 1057157 / 517
β₂ (95% CI)0.236 [0.209, 0.263]0.730 [0.686, 0.772]0.871 [0.818, 0.919]0.340 [0.311, 0.370]0.756 [0.726, 0.786]0.381 [0.357, 0.406]
β₂ Seeds / Evolved Instances198 / 1896176 / 1354189 / 737193 / 1878192 / 1614199 / 2154

Q2: Random Seeds

We have added experiments with three random seeds (42, 30, and 40). We used the same checkpoint-selection rule for all runs: all models were trained for one epoch, and the final checkpoint was used for evaluation.

In the manuscript, we describe clean training as follows (Line 391): “the gains can be marginal and may even induce performance decrease.” For mixed training, we state (Line 410) that it “provides a more balanced improvement across workflows: although it is not always the best, its performance is generally close to top-1.”

We report the performance degradation under clean training and the gains under mixed training across the random seeds, relative to the no-training baseline.

TrainingRecovery Δ Acc. (pp)Reuse Δ Acc. (pp)
Clean−12.14 ± 0.90−4.06 ± 1.29
Mixed+30.41 ± 12.11+20.60 ± 2.10

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

We sincerely thank all reviewers for their thoughtful feedback and recognition of our contributions. We are encouraged by their positive assessments of:

  1. An effective and well-controlled task generation framework, evolving linked tasks from shared seeds and enabling instance-level comparisons across workflows (yJrW, 9UFQ, tsp4, Jhct).

  2. Practical value of the empirical findings for reasoning across workflows and corrupted contexts, revealing shifts in model rankings and potential performance degradation from clean-only training (yJrW, tsp4, Jhct).

  3. Valuable insights into model behavior and training, including bidirectional success/failure transitions, role anchoring, and zero-shot transfer from mixed training to downstream reuse (yJrW, tsp4, Jhct).

  4. Robust experimental validation, with a series of robustness checks supporting the main findings and additional experiments providing exploratory evidence (yJrW).

We also appreciate the reviewers' constructive suggestions and have addressed their concerns in our rebuttal:

  1. Provided additional analyses and illustrative examples for clearer interpretation, including direct Mode-1-to-Mode-3 transition rates, the executed recovery-to-reuse pipeline (tsp4, Jhct, yJrW).

  2. Added further experimental evidence to validate our findings, including expanded seed-task evaluations, one-shot ablations, and out-of-domain generalization analyses (yJrW, 9UFQ, tsp4, Jhct).

  3. Clarified evaluation and data-generation details, including evaluation prompts, task distributions, and synthetic-data quality checks (9UFQ, tsp4, Jhct).


During the rebuttal period, reviewer Jhct acknowledged that our additional experiments and clarifications addressed their main concerns and raised their rating to a positive recommendation. Furthermore, reviewer tsp4 maintained their original positive recommendation.

(Due to PDF updates, some line and figure references in our earlier rebuttal may not align with the current version. We have included the previous PDF in the Supplementary Material for reference.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.