acceptodds
Under review as a conference paper at ICLR 2027

SC-CoIT: Self-Corrective Interleaved Chain of Thought via Bidirectional Cross-Modal Recovery

Abstract

Unified multimodal models (UMMs) can generate Interleaved Chain of Thought (Interleaved CoT) traces that alternate between language and visual thoughts. Yet correct-only supervised fine-tuning provides no recovery target when an intermediate node fails. We identify two cascade failures: a Draw Error, where the text expert proposes a valid action but the image expert misexecutes it, and a Policy Error, where the text expert proposes an action inconsistent with the task or environment and the image expert follows it without verification. From ground-truth Interleaved CoT traces, we construct a bidirectional error-recovery training dataset and propose (SC-CoIT), which trains the modality experts to check each other's outputs and recover from cross-modal errors. For Draw Errors, the text expert grounds the discrepancy and directs a same-step redraw; for Policy Errors, the image expert emits a visual reflection signal, prompting the text expert to regenerate the step. To adapt both recovery branches to failure modes that evolve during training, iterative error replay samples new errors from the current checkpoint on validated prefixes and adds them to subsequent training. Together, these mechanisms improve the model's ability to inspect generated visual states, reject invalid actions before execution, and recover from both error types within the same Interleaved CoT trajectory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified multimodal models, think with images, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8tt4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SC-CoIT studies error recovery in interleaved text-image reasoning. Starting from ground-truth Zebra-CoT trajectories, the method samples model-generated errors and constructs recovery examples for two failure types: valid textual actions that are incorrectly rendered, and invalid textual actions that require correction. Textual reflections guide image recovery, while a visual rejection signal prompts textual recovery; the corrected nodes reconnect to the original reference trajectory. Iterative error replay refreshes the training examples using successive checkpoints. Built on BAGEL-Zebra-CoT, the method reports improved performance on VisWorld-Eval and analyzes error detection and recovery behavior.

Critical strengths
  1. Learning to recover from intermediate errors is an interesting and practically relevant direction for interleaved multimodal reasoning.
  2. The two complementary recovery branches provide a meaningful approach to correcting both textual and visual mistakes.
  3. Substantial gains on VisWorld-Eval, branch-specific ablations, and detection-and-repair measurements provide encouraging empirical evidence.
Critical weaknesses
  1. The coarse error taxonomy and lack of analysis of scenario-specific failures, compound errors, and recovery rates by subtype leave the transferability of the learned correction mechanism unclear.
  2. How are complete recovery trajectories validated to ensure that VLM-generated reflections agree with ground-truth recovery nodes and remain consistent with subsequent reasoning, and how are inconsistent samples handled?
  3. Please count how many tasks were actually done correctly because of reflection. Did the performance improve because of reflection or because more rounds of continued training were done?
Questions for authors
  1. What evidence isolates the contribution of explicit reflection from additional training and error-recovery supervision without reflection?
  2. What validation evidence shows that VLM-generated reflections agree with the ground-truth recovery nodes and remain consistent with the subsequent trajectories?
4 comments
AuthorsOpenReview ↗

Response to Questions1

To separate the effects of explicit reflection, error recovery supervision without reflection, and extra training, we added two experiments and report the comparison below.

First, we trained a model without explicit reflection. For Draw Errors, we replaced the reflection text with a label that only indicates whether the image is correct. For Policy Errors, we replaced the red-x generation task with simple binary classification using the Reflect Head described in the paper. Without explicit error diagnosis, the model improves only slightly over the single-branch Reflect Head. This shows that adding Draw feedback that only judges image correctness brings a small gain on top of binary Policy gating. Binary feedback can trigger a retry, but it does not say what needs to be repaired. Redrawing is therefore closer to resampling under the original conditions. Without specific correction conditions, extra sampling in the continuous space has more difficulty recovering the visual evidence needed for later reasoning than sampling in the discrete text space.

We also ran an extra-training experiment. We used only correct training trajectories and trained for the same number of extra steps. The model shows overfitting. This indicates that extra training alone does not bring gains.

Table 1: Comparison of extra training, negative-sample supervision, and the full model

MethodPaper FoldingMulti-HopBall TrackingCubeMMSIAvg-5
BAGEL-Zebra-CoT11.6722.9212.1122.2927.2019.24
Extra training on correct trajectories only5.4221.0415.9225.0019.9217.46
Reflect Head (Table 4)17.0832.0830.8633.9632.7629.35
Negative-sample supervision (no explicit reflection)18.7534.1732.0335.8334.1030.98
SC-CoIT (full model)28.7542.2929.8850.0049.0439.99
AuthorsOpenReview ↗

Response to Questions2

Appendix A.1 provides the inputs and prompt used by the VLM to generate reflection. As shown there, the VLM sees the current instruction, the previous correct state, the incorrect image, and the GT image for the current step at the same time. The prompt asks the VLM to keep the language and style of the original reasoning trajectory, point out the specific error, and state the corrections needed to recover the GT state. Since the VLM uses the GT trajectory as a reference, its reflection is consistent with the later trajectory in most cases. We also added annotated examples in Appendix A.1 (Figures 5 and 6) to further show this consistency.

AuthorsOpenReview ↗

a gentle follow-up on our response above

We wanted to follow up on the comparisons and examples in our reply. Have the extra-training and binary-feedback comparisons helped clarify what explicit reflection adds? We also included examples in Appendix A.1 to show how reflections match the recovery steps. We would appreciate hearing whether these are helpful and what still needs clarification.

The authors added extra-training and no-reflection comparisons and qualitative examples, but my concerns remain unresolved because the response lacks systematic validation of recovery trajectories, clarification of how inconsistent samples are handled, and error-subtype analyses; I therefore maintain my original score.

Reviewer_ccQSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Unified multimodal models can solve problems through interleaved chains of thought that alternate between textual reasoning and intermediate visual states. However, supervised fine-tuning exclusively on correct trajectories leaves models unprepared for erroneous contexts at inference time, resulting in cascading failures from drawing errors and policy errors.

The paper proposes SC-CoIT, which incorporates cross-modal checking and recovery into training through two branches. Reflect-Draw-SFT trains the text expert to inspect an incorrect image, identify visual discrepancies, and request a redraw within the same reasoning step. Reflect-Policy-SFT trains the image expert to reject invalid textual actions by producing RedX, a predefined visual signal showing a red cross on a gray background, prompting the text expert to generate a valid action.

The framework also incorporates iterative error replay across three training rounds. Newly observed errors are sampled from the current checkpoint and added to the training data, allowing recovery supervision to adapt to evolving failure modes.

Critical strengths
  1. The two recovery branches effectively use collaboration between the text and image experts. Reflect-Draw-SFT enables the text expert to identify and correct visual execution errors, while Reflect-Policy-SFT enables the image expert to reject invalid textual actions and trigger replanning. This design makes constructive use of the complementary roles of the two modalities.
  2. The loss masking design is well motivated. Directly supervising erroneous nodes would encourage the model to reproduce errors, while removing them would deprive it of the context needed to learn recovery. The authors assign zero generation loss to erroneous nodes through binary masks at the node level, while retaining these nodes in the attention context for subsequent reflection and retry steps. This allows the model to learn from errors without being trained to generate them.
  3. The experimental results show substantial improvements on BAGEL, demonstrating stronger reflection and error recovery capabilities.
Critical weaknesses
  1. Although iterative error replay is described in the main text and Appendix A.2, the paper lacks quantitative or qualitative analysis demonstrating its effectiveness.
  2. The offline VLM used for data construction is not identified. Please specify the model and explain how potential hallucinations in its outputs are handled. If the recovery data are constructed solely using this VLM, I would expect the quality and scope of the correction supervision to be constrained by its capabilities.
  3. Table 3 reports an error detection precision of 97.47, but a Repair/Det. score of only 4.80, whereas Table 1 shows substantial improvements from SC-CoIT. Please explain why the repair rate among detected errors is so low. Given that more than 91% of detected errors are not successfully repaired at the next node, what accounts for the substantial end-to-end gains in Table 1?
Questions for authors

Please address the questions and missing comparisons raised in the Critical Weaknesses section.

6 comments
AuthorsOpenReview ↗

Response to Questions1

We added an ablation without iterative error replay and compared it with the full model already reported in the paper. Both settings use the same training budget. Without replay, Overall drops from 29.07% to 25.98%, a decrease of 3.09 percentage points. Avg-5 drops from 39.99% to 35.84%, a decrease of 4.15 percentage points. Accuracy drops on all five main tasks. Cube 3-View and MMSI drop by 5.54 and 4.37 percentage points, respectively. These results show that replay further improves final task accuracy on top of the two-branch recovery training. In each round, we collect new errors from the current model and mix them with past errors for further training. This ablation adds quantitative evidence for the benefit of replay.

Table 1

VisWorld-Eval metricWith replayWithout replay
Paper Folding28.7525.33
Multi-Hop Manipulation42.2938.42
Ball Tracking29.8826.33
Cube 3-View50.0044.46
MMSI Positional Relationship49.0444.67
Maze0.000.00
Sokoban0.830.66
Avg-539.9935.84
Avg-728.6925.70
Overall29.0725.98
AuthorsOpenReview ↗

Response to Questions2

The VLM used for offline data construction is Qwen3.7-Plus. To prevent its possible hallucinations from reducing the quality of the recovery data, corrected text, recovery images, and later trajectory steps are all taken directly from the original ground-truth (GT) nodes. The offline model only detects errors and writes reflection annotations. It does not generate correction paths.

Thus, expert GT trajectories ensure the quality of the supervision data. We also provide the original question, final answer, current step, previously verified GT context, and images as input references. This reduces the need to guess the correct state. At inference time, the model generates reflection and recovery steps on its own based on the question and images.

AuthorsOpenReview ↗

Response to Questions3

The single-node repair rate in the paper uses a strict standard. A recovered node must have no errors in image quality, text understanding, or reasoning. The end-to-end gains come from two sources: better initial reasoning accuracy and practical gains from fixing key local errors at inference time. In actual reasoning, a correct final answer does not require every part of a repaired node to be perfect. A local correction can resolve the key issue that blocks progress. For example, an image can restore the key elements named in the reflection, or a text node can correct the direction of the final action. This provides useful information for later steps and can lead to the correct answer through multiple reasoning steps.

To measure this contribution, we added sample-level and relaxed node-level repair metrics in Tables 1 and 2. Under the relaxed standard, an image must meet the reflection requirements, and a text node must give the correct final action. The text and image node-level repair rates then rise to 21.15% and 10.58%, respectively. At the sample level, a trajectory can contain multiple error nodes, while the final answer is judged for the whole sample. The image and text branches reach sample-level repair rates of 37.74% and 24.88%, respectively. Among samples with correct final answers, 82.36% and 39.65% trigger reflection and correction in the image and text branches, respectively. These results show that local correction at inference time makes a real contribution to final accuracy, even when a single node does not meet the strict fully correct standard.

Table 1: Sample-level metrics

TaskImage: sample-level repair rateImage: detection rate among correct samplesText: sample-level repair rateText: detection rate among correct samples
Ball Tracking29.91%100.00%20.48%50.00%
Cube49.69%96.43%31.68%41.67%
MMSI48.00%46.67%18.24%20.69%
Multi-Hop42.20%100.00%37.47%56.67%
Paper Folding23.30%75.00%15.69%33.33%
All five tasks37.74%82.36%24.88%39.65%

Table 2: Relaxed node-level repair metrics

TaskImage: strict repair rateImage: relaxed repair rateText: strict repair rateText: relaxed repair rate
Ball Tracking1.37%9.34%6.67%10.37%
Cube1.71%7.12%11.82%24.55%
MMSI17.35%40.82%7.08%23.01%
Multi-Hop8.08%10.51%14.89%35.11%
Paper Folding2.58%5.15%4.72%16.98%
Overall4.80%10.58%8.78%21.15%
Reviewer_JKR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes SC-CoIT, a self-correcting interleaved chain-of-thought framework for unified multimodal models. The method distinguishes two types of intermediate failures: Draw Errors, where generated images fail to execute valid textual instructions, and Policy Errors, where textual actions are inconsistent with the current task state or environment. SC-CoIT constructs bidirectional recovery trajectories to train text and image experts to detect and correct each other's errors, complemented by iterative error replay to adapt to evolving failure patterns. Experiments on VisWorld-Eval report substantial improvements over the BAGEL-Zebra-CoT baseline. However, the contribution of actual inference-time recovery to the reported gains remains insufficiently established, and the effectiveness of the proposed recovery mechanisms requires further investigation.

Critical strengths
  1. Interesting problem formulation. The paper addresses error propagation in interleaved multimodal reasoning, where intermediate textual and visual mistakes can affect subsequent reasoning steps. Distinguishing Draw Errors from policy errors offers a meaningful perspective on cross-modal error recovery.
  2. Structured bidirectional recovery framework. SC-CoIT introduces complementary recovery branches that allow text and image experts to inspect and correct each other's outputs. The incorporation of iterative error replay further attempts to adapt recovery supervision to the model's evolving failure patterns.
  3. Promising results and detailed analyses. The method substantially improves overall accuracy over the underlying baseline, from 13.66% to 29.07%. The paper also includes branch-level ablations, error detection and repair evaluations, reflection representation comparisons, attention analyses, and both successful and unsuccessful recovery examples.
Critical weaknesses
  1. Insufficient attribution of accuracy gains to actual self-correction. Although the authors explicitly acknowledge the low immediate repair rates (4.80% for Draw Errors and 8.78% for Policy Errors) and provide qualitative analyses of unsuccessful recovery cases, the paper does not adequately explain how these limited repair capabilities relate to the substantial overall accuracy improvement. It remains unclear whether the gains primarily arise from successful inference-time recovery, improved initial reasoning and generation, or other effects of recovery-oriented training.

  2. Potential inconsistency between the Policy Error definition and the evaluated recovery behavior. The paper formally defines Policy Errors as locally unexecutable actions under the current task state and constraints. However, the successful Policy reflection example in Figure 8 appears to involve an incomplete reasoning or computation step rather than an unexecutable action. This raises concerns about whether the proposed error taxonomy is consistently applied during data construction, evaluation, and qualitative analysis.

Questions for authors
  1. Recovery effectiveness: Can the authors disentangle improvements in initial reasoning accuracy from gains due to successful inference-time recovery? In particular, how much does overall accuracy change when recovery is disabled at inference while keeping the trained model unchanged, and does SC-CoIT reduce the frequency of initial Draw and Policy Errors compared with the baseline?

  2. Error taxonomy: How is the incomplete computation in Figure 8 classified as a Policy Error under the paper's definition of locally unexecutable actions? Are incomplete reasoning steps or other errors beyond locally unexecutable actions included in the training and evaluation annotations?

5 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 & Questions 1(part1)

1. Repair rates and final-answer accuracy

The reviewer first asked how the limited repair capabilities relate to the substantial overall accuracy improvement. We clarify that the repair rate reported in our paper is calculated using a relatively strict criterion. To further investigate how the repair rate affects our method’s performance, we provide additional experimental statistics focusing on two aspects: (1) relaxing the criterion for counting an individual node as repaired, and (2) quantifying the proportions of repaired nodes under different conditions. We discuss these results in detail below:

(1) A repaired node must have no errors in image quality, text understanding, or reasoning. This leads to a low repair rate in our original paper. Actually, a correct final answer does not require the whole node to be fully correct. For example, fixing the key image elements named in the reflection, or producing the correct final action in a text node, can help the reasoning trajectory. We therefore provide a comparison using relaxed node-level repair metrics in Table 1. For images, we check whether the repair meets the requirements in the reflection text. For text, we check whether the final action is correct. These relaxed metrics more clearly show how successful correction at inference time contributes to accuracy.

Table 1: Relaxed node-level repair metrics

TaskImage: strict repair rateImage: relaxed repair rateText: strict repair rateText: relaxed repair rate
Ball Tracking1.37%9.34%6.67%10.37%
Cube1.71%7.12%11.82%24.55%
MMSI17.35%40.82%7.08%23.01%
Multi-Hop8.08%10.51%14.89%35.11%
Paper Folding2.58%5.15%4.72%16.98%
Overall4.80%10.58%8.78%21.15%

Although the relaxed image repair rate remains low, it is not directly proportional to final-answer accuracy: a partial repair can still provide key visual information for later steps. To further explain the link between repair behavior and final results, we added specific case analyses to the CASE STUDIES section of the appendix in the revised paper. These cases show several common patterns linking repair behavior to correct answers.

(2) To separate gains in initial reasoning accuracy from gains due to successful recovery at inference time, we provide sample-level metrics in Table 2. For each task, we report the share of correct samples that contain reflection in each modality. We also report the sample-level repair rate, where the model detects an error and gives the correct final answer. Both image and text reflection occur in a share of correct samples. In most settings, image reflection mainly brings gains from successful recovery at inference time. Text reflection brings both better initial reasoning accuracy and gains from successful recovery at inference time.

Table 2: Sample-level metrics

TaskImage: sample-level repair rateImage: detection rate among correct samplesText: sample-level repair rateText: detection rate among correct samples
Ball Tracking29.91%100.00%20.48%50.00%
Cube49.69%96.43%31.68%41.67%
MMSI48.00%46.67%18.24%20.69%
Multi-Hop42.20%100.00%37.47%56.67%
Paper Folding23.30%75.00%15.69%33.33%
All five tasks37.74%82.36%24.88%39.65%
AuthorsOpenReview ↗

Response to Weaknesses 1 & Questions 1(part2)

2. Recovery effectiveness

The reviewer also asked about the improvement gains due to successful inference-time recovery.
Each modality expert uses its own output space to decide whether to continue reasoning or reflect and correct, based on the previous modality's output. We therefore cannot explicitly disable recovery at inference time.

To further show the benefits of reflection training, we added two ablations (Table 3). In the first, Negative-sample supervision (no explicit reflection), we replaced Draw reflection text with a correct/incorrect label and Policy red-x generation with Reflect Head binary classification. Binary feedback still helps text reasoning, but image feedback without explicit reflection brings only a small gain. In the continuous image space, resampling without a correction target may fail to recover the visual evidence needed for later reasoning. In the second, we trained on correct trajectories for the same number of extra steps. The model overfits, showing that extra training alone does not bring gains.

Table 3: Training ablations and the full model

MethodPaper FoldingMulti-HopBall TrackingCubeMMSIAvg-5
BAGEL-Zebra-CoT11.6722.9212.1122.2927.2019.24
Extra training on correct trajectories only5.4221.0415.9225.0019.9217.46
Reflect Head (Table 4)17.0832.0830.8633.9632.7629.35
Negative-sample supervision (no explicit reflection)18.7534.1732.0335.8334.1030.98
SC-CoIT (full model)28.7542.2929.8850.0049.0439.99
AuthorsOpenReview ↗

Response to Questions2

In our annotation scheme, the error in Figure 8 is a Policy Error caused by an unmet precondition. The task asks for the difference between the number of upward-pointing triangles and circles. Before reflection, the model gave a plan but skipped the actual counting step. The later subtraction therefore lacked the required intermediate values, making it a locally unexecutable action. We classify it as a Policy Error caused by an unmet precondition. After reflection, the model counted 6 triangles and 1 circle, then completed the calculation: 6−1=56-1=5.

The training and evaluation annotations also include this category. We judge whether the current node completes its required reasoning and provides the information needed for later steps. A short but complete node is acceptable. However, a node that states an intention but skips a required operation is included in recovery supervision.

To avoid ambiguity, we also replaced the successful and failed Policy reflection examples in the revised paper (Figures 15 and 16). We chose examples with clearer error categories to show the difference between successful correction and error propagation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.