- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Evolutionary Task Generator (ETG), a lineage-preserving and graph-annotated framework for studying LLM reasoning across three workflow modes: direct solving, explicit recovery from a corrupted interaction, and downstream reuse of the recovered state. Starting from a common synthetic reasoning graph, ETG constructs six corruption types by crossing two corruption targets (value and dependency) with three interaction regimes (static corruption, unsupported assumption, and evidence-integration failure). The paper further defines two instance-level transition metrics: , measuring failure-to-success transitions between consecutive modes, and , measuring success-to-failure transitions.
Critical strengths
- Constructing Solve, Recover, and Reuse tasks from the same seed problems enables instance-level comparisons across workflows and corruptions, revealing that strong direct-solving performance does not guarantee reliable recovery or downstream performance under corrupted contexts. This provides a more informative evaluation of LLM reasoning.
- The experiments show that clean-only fine-tuning can improve direct-solving accuracy while degrading recovery and downstream performance, whereas mixed training generally yields more balanced improvements. These findings highlight the importance of evaluating training strategies across workflows rather than relying solely on direct-solving gains.
Critical weaknesses
- Although the paper explicitly treats failure-to-success transitions as descriptive rather than causal, these transitions do not establish whether models actually leverage useful information from the corrupted history or simply ignore it and solve the task again. The repeated claim of "resist the harmful and leverage the beneficial" lacks direct experimental support.
- Mode 3 only evaluates corrupted histories, without comparing against correct histories or no prior history on the same downstream tasks. This makes it difficult to determine whether models can selectively identify reliable information or simply distrust prior answers.
Questions for authors
- In Table 2,Claude-Opus-5 achieves 0.98 accuracy on 200 Mode 1 seed tasks,leaving only four initially failed seeds.Since α₁ is derived from these four seed-task lineages,how statistically reliable is the reported high α₁?
- The paper shows performance differences across workflow modes and corruption types,but the main contributing factors remain unclear.For example,why are unsupported assumptions harder to recover from?What are the relative impacts of corruption target, interaction regime, turns or roles?Could the authors clarify these factors and provide insights for future model evaluation and training?
- Mode 3 constructs downstream tasks using a simple linear relation,y₂=f(y₁).Would the paper's main findings still hold for downstream tasks involving more complex,multi-step dependencies?
- Could the authors report the counts and proportions of all six corruption types in mode?Are the overall results driven by particular types?Please also provide a few paired failure-to-success and success-to-failure examples to show what changes in the model’s responses.
6 comments
Response to Reviewer Jhct (2/4)
W2: Mode-3 Ablation Study
We thank the reviewer for suggesting insightful experiments on Mode 3.
We selected the instances models answered correctly in Mode 3 and evaluated those same instances without context. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.
| Model | Mode-3 Accuracy | No-Context Mode-3 Accuracy |
|---|---|---|
| GPT-4o | 1.000 | 0.788 |
| GPT-5.4 | 1.000 | 0.836 |
The no-context condition serves as a proxy for ignoring the context. Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information in the added contexts.
Below is a concrete example from DeepSeek’s output in which the model explicitly corrects the upstream calculation and uses the corrected result to answer the downstream question, rather than simply ignoring the context:
Correcting the earlier calculation: Fashion Backpack has 6 Text Markers and the same number of Highlighters, so 6 Highlighters.
Total stationeries: .
Dollars: .
The final answer is <<16>>.
Response to Reviewer Jhct (3/4)
Q1
Claude's very high accuracy in Mode 1 left relatively few failure cases available for transition analysis. To expand this analysis, we identified 24 tasks that Claude answered incorrectly among 1,300 Mode-1 tasks and generated 795 Mode-2 instances from these seed tasks.
Our conclusion remains unchanged: beneficial transitions are non-negligible, with α₁ = 0.634 (95% CI: [0.523, 0.738]).
For α₁, we also report the estimates and 95% CIs for the other models, together with the numbers of Mode-1 seed tasks and corresponding evolved Mode-2 instances.
| GPT-5.5 | GPT-5.4 | GPT-4o | Gemini-3.6-Flash | DeepSeek-v4-Flash | |
|---|---|---|---|---|---|
| α₁ (95% CI) | 0.444 [0.359, 0.533] | 0.194 [0.151, 0.240] | 0.232 [0.208, 0.257] | 0.339 [0.248, 0.430] | 0.343 [0.280, 0.409] |
| Seeds / Evolved Instances | 22 / 333 | 82 / 1223 | 102 / 1496 | 36 / 549 | 50 / 747 |
Q2
The role ablation provides further insight into cases of unsupported assumptions. (Line 303) When we change the role introducing the unsupported assumption from Assistant to User, accuracy improves substantially. In some settings, accuracy becomes comparable to that in other Mode-2 settings. As discussed in Line 303, the lower performance in the User-Assistant configuration suggests stronger anchoring on the assistant’s false beliefs.
We also report average accuracy grouped by several factors below.
| Factor | Category | Mode-2 Acc. |
|---|---|---|
| Corruption Target | Value (Node) | 0.6015 |
| Corruption Target | Dependency (Edge) | 0.5997 |
| Interaction Regime | Static Corruption | 0.7499 |
| Interaction Regime | Unsupported Assumption | 0.4032 |
| Interaction Regime | Evidence-Integration Failure | 0.6491 |
| Turns | Single-turn | 0.7499 |
| Turns | Multi-turn | 0.4920 |
Among these grouped averages, interaction regime exhibits the largest performance gap, with Unsupported Assumption yielding the lowest accuracy. The difference between corruption targets is relatively small. Multi-turn accuracy is substantially lower than single-turn accuracy.
These findings highlight the importance of extending model evaluation and training beyond idealized clean contexts to include corrupted contexts, particularly unsupported assumptions introduced by the models themselves in multi-turn dialogues.
Q3
We extended our Mode-3 evaluation to two-step downstream tasks, and our main findings continue to hold.
| Model | Acc. | Misled Rate | α₂ | β₂ |
|---|---|---|---|---|
| GPT-4o (1-step) | 0.0636 | 0.9784 | 0.0388 | 0.8711 |
| GPT-4o (2-step) | 0.0535 | 0.9936 | 0.0331 | 0.8928 |
| GPT-5.4 (1-step) | 0.1733 | 0.9520 | 0.0744 | 0.7304 |
| GPT-5.4 (2-step) | 0.0824 | 0.9886 | 0.0296 | 0.8663 |
Models remain susceptible to being misled. Compared with one-step tasks, two-step tasks yield lower accuracy and a higher misled rate among parseable errors.
Response to Reviewer Jhct (1/4)
We greatly appreciate your recognition of our informative reasoning evaluation and the practical implications of our training findings. Thank you for your thoughtful suggestions on strengthening the analysis and experimental evidence. We address your concerns below.
W1: Further Evidence for Transitions
Mode 1 provides a no-context reference for the idealized case of “simply ignoring the corrupted context.”
Specifically, α₁ measures the proportion of Mode-2 instances answered correctly among those evolved from seed tasks that the model answered incorrectly in Mode 1. These transitions therefore capture cases where the model fails without context but succeeds with corrupted context. Table 4 reports α and β; for example, GPT-5.5 achieves α₁ 0.444.
We also selected the instances models answered correctly in Mode 3 and evaluated those same instances without context. Mode-3 accuracy on this selected subset is therefore 1.0 by construction.
The no-context condition serves as a proxy for ignoring the context. Removing the context reduces accuracy from 1.0 to 0.788 for GPT-4o and to 0.836 for GPT-5.4. These results suggest that at least some successful Mode-3 responses benefit from useful information in the added contexts.
Regarding training, Figure 4 shows that mixed training increases α and decreases β. Here, α captures beneficial failure-to-success transitions, where the model answers incorrectly in the preceding mode but correctly in the current mode. Conversely, β captures harmful success-to-failure transitions, where the model answers correctly in the preceding mode but incorrectly in the current mode.
Building on these findings, the phrase “suggest a more constructive direction” in our abstract (Line 26) is intended to introduce a future research direction: