acceptodds
Under review as a conference paper at ICLR 2027

ClueJigsaw: Learning to Reason with Visual Clues through Jigsaw Reconstruction

Abstract

Visual reasoning is essential for advancing vision-language models (VLMs) toward a deeper understanding of the visual world. However, perceiving and interpreting visual inputs remains challenging when familiar spatial structures are disrupted, as exemplified by jigsaw reconstruction. We find that successful reconstruction benefits from the joint use of global semantics and local structural clues. Motivated by this finding, ClueJigsaw is proposed as a post-training framework for clue-aware jigsaw reconstruction. We introduce a clue-guided trajectory construction method and use SFT cold-start training to develop models’ ability to independently infer semantic clues and structural clues from shuffled inputs. Clue-aware reinforcement learning is then introduced through an easy-to-hard curriculum of reconstruction subtasks to progressively improve clue identification and utilization. We design a two-part reward that combines answer correctness with VLM-based assessment of clue use, additionally rewarding reasoning grounded in appropriate visual clues. Experiments show substantial improvements on both and jigsaw tasks, with reconstruction accuracy rising from 10.12% to 97.38%, alongside gains on general visual reasoning benchmarks. These results highlight the value of visual clue supervision and verification in VLM post-training, helping models connect visual understanding with clue-grounded reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Visual reasoning, Jigsaw reconstruction, Multimodal post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CqFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes clue-aware RL for visual jigsaw puzzles using structural and semantic clues. Through training with clues, the model performs better on visual jigsaw tasks. Experimental results show that ClueJigsaw also benefits performance across various visual benchmarks.

Critical strengths
  • The motivation for using clues is clear and intuitive.

  • Training on ClueJigsaw alone transfers to general visual benchmarks.

Critical weaknesses
  • Missing comparison with standard jigsaw training such as [1, 2]

  • It is unclear whether clues help general visual tasks. Does the model perform better than jigsaw training alone?

Reference:

[1] Visual Jigsaw Post-Training Improves MLLMs, ICLR 26

[2] Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzles, TMLR 26

Questions for authors

Does the model also train on standard RL data?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for helping strengthen our paper and clarify the baseline comparisons and training data. We hope these responses address your concerns and would appreciate your reconsideration of the score.

Q1. Benefits Beyond Jigsaw Training Alone

To address W1, we compare reconstruction performance with Visual Jigsaw (Wu et al., 2026) and Jigsaw-R1 (Wang et al., 2025), retraining both methods with the same backbone and shared reconstruction data and evaluating their released checkpoints as supplementary baselines. To address W2, we extend these comparisons to the six general visual benchmarks to examine whether clue-guided training provides additional transfer benefits beyond jigsaw training alone.

Our primary comparison retrains both approaches using the same Qwen3-VL-8B-Instruct backbone and shared full-restoration training data. These baselines optimize answer correctness and output format without explicit clue supervision or clue-use rewards. We additionally evaluate the authors' released checkpoints. The experimental settings are detailed below.

  • Visual Jigsaw. Following the original method, we retrain Visual Jigsaw-8B* using GRPO without an SFT cold start. The accuracy reward is 1 for exact reconstruction, 0.2 times the correct-position fraction for other valid permutations, and 0 for invalid ones. Correct formatting earns an additional 0.2, and KL regularization is disabled. We also directly evaluate the released Visual Jigsaw-7B checkpoint.

  • Jigsaw-R1. Following the original method, we retrain Jigsaw-R1-8B* using GRPO without an SFT cold start. The accuracy reward is the correct-position fraction, with an additional format reward of 0.5 and a KL coefficient of 0.04. We also directly evaluate the released Thinking 3B checkpoint, denoted as Jigsaw-R1-3B.

The following table reports reconstruction accuracy and general visual benchmark scores (%), with bold and underlining indicating the best and second-best results, respectively.

\hline \text{Setting} & \text{Method} & \rlap{\text{Jigsaw}} & & \rlap{\text{General Visual Benchmarks}} & & & & & \\\\ & & 2\times2 & 3\times3 & \text{MME} & \text{RWQA} & \text{MMVP} & \text{BLINK} & \text{Hall} & \text{MMMU} \\\\ \hline \text{Released} & \text{Visual Jigsaw-7B} & 55.4^\dagger & 41.4^\dagger & 46.7 & 65.2 & 52.0 & 55.0 & \underline{65.4} & 52.1 \\\\ & \text{Jigsaw-R1-3B} & 0.4 & 0.0 & 45.0 & 60.8 & 39.3 & 47.0 & 59.8 & 43.2 \\\\ \hline \text{Retrained} & \text{Visual Jigsaw-8B}\ast & \underline{71.2} & \underline{51.7} & \underline{48.5} & 67.3 & \underline{60.0} & \underline{59.9} & 61.5 & 52.8 \\\\ & \text{Jigsaw-R1-8B}\ast & 44.8 & 20.6 & 48.2 & \underline{69.5} & 59.3 & 59.3 & 60.7 & \underline{53.4} \\\\ \hline \text{Ours} & \mathbf{ClueJigsaw\text{-}8B} & \mathbf{97.4} & \mathbf{64.1} & \mathbf{50.3} & \mathbf{71.2} & \mathbf{66.0} & \mathbf{62.1} & \mathbf{67.6} & \mathbf{56.9} \\\\ \hline \end{array}$$ $^\dagger$ Since some test source images appear in Visual Jigsaw's public training data, we report its $2\times2$ and $3\times3$ accuracies on the non-overlapping subset. **Reconstruction Performance.** ClueJigsaw achieves higher exact reconstruction accuracy than the jigsaw-specific baselines, supporting the effectiveness of our framework relative to existing jigsaw post-training approaches. **Additional Transfer Benefits.** ClueJigsaw also achieves gains on general visual reasoning benchmarks over correctness-only jigsaw training, suggesting that our clue-guided post-training framework provides additional transferable benefits beyond jigsaw training alone. ### Q2. Post-Training Data Our additional SFT and RL stages use only jigsaw-related data, without mixing in general-purpose RL or VQA data. The six general visual benchmarks are used solely for evaluation, with no benchmark-specific fine-tuning. The resulting gains highlight the transfer benefits of clue-guided jigsaw post-training without additional general-purpose training data.

As the discussion period draws to a close, we would like to check whether our response has adequately addressed your concerns. We would be happy to clarify any remaining points while there is still time.

With about 24 hours left in the discussion period, we would appreciate your feedback on our response. Please let us know if any concerns remain. Thank you!

Reviewer_8HWnOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ClueJigsaw trains a vision-language model to solve image jigsaw puzzles in two stages. In the SFT warm-up, a larger teacher model sees the intact image and writes semantic clues (what the scene shows) and structural clues (edges, text, and colors that continue across patches). It uses these clues to write step-by-step solutions. The student learns to produce clues, reasoning, and an answer from the shuffled patches alone. In the RL stage, the student practices full reconstruction and four easier subtasks, with a reward that combines answer correctness and a judge's score for clue use. Accuracy rises from 10.1% to 97.4% on 2 by 2 puzzles and from 0.6% to 64.1% on 3 by 3 puzzles, with gains on six general visual benchmarks (generalization) .

Critical strengths
  1. The method is simple and also working. It uses standard parts: teacher-written training data, filtering, and RL with DAPO. The reward is sensible: partial credit for each correct patch, most weight on a fully correct answer, and a small bonus for good clue use. The paper also tests its own idea: training with both clue types reaches 97.4% on 2 by 2 puzzles, versus 61.3% without clues (Table 3, p. 8).

  2. The model has large gains on the task. The 8B model reaches 64.1% on 3 by 3 puzzles, where Qwen3-VL-235B gets 3.3% (Table 1, p. 6). It also improves on all six general benchmarks, including MMVP (59.3% to 66.0%) and HallusionBench (59.5% to 67.6%). No intact image or judge is needed at test time.

  3. The project has careful evaluation and documentation. The test set uses 300 COCO images held out from training and 4,800 puzzles balanced by how many patches are displaced, so results are reported from mild to heavy shuffling (Appendix A.2). A puzzle counts as solved only if the whole order is correct. All prompts are in the appendix, and at test time the model needs no intact image or judge.

Critical weaknesses
  1. In the no-clue variant, the teacher also loses the hints from the intact image (p. 7), so it likely produces fewer or worse training examples. The gap from 61.3% to 97.4% may reflect training-data quality rather than the student's use of clues.
  2. Visual Jigsaw (Wu et al., 2026) already reports benchmark gains from jigsaw post-training, and Jigsaw-R1 applies rule-based RL to puzzles. Both are cited but not compared. Without them, the benchmark gains could come from jigsaw training itself rather than from clues.
  3. The base model scores below its official numbers. The Qwen3-VL technical report (Bai et al., 2025, Table 4), which the paper cites, lists Qwen3-VL-8B-Instruct at 69.6 on MMMU, 69.1 on BLINK, and 71.5 on RealWorldQA. In Table 2 (p. 7), the base model scores 53.3, 59.9, and 69.5. Even the final model (56.9, 62.1, and 71.2) stays below the official base numbers. Part of the reported transfer gains may therefore come from the evaluation setup rather than from training.
Questions for authors
  1. Additional evidence: With the same base model and data, how does ClueJigsaw compare with correctness-only jigsaw RL (as in Jigsaw-R1 or Visual Jigsaw), on both puzzles and the six benchmarks? I ask because Visual Jigsaw already reports benchmark gains from jigsaw training without clues, so this comparison is needed to credit the gains to clues.
  2. Additional evidence: How many training examples does each Table 3 variant keep after filtering? Does the gap remain if every variant's teacher uses the intact-image hints, and only the clue text is removed from the student's training targets? I ask because the teacher in the no-clue variant loses the intact-image hints (p. 7), so the gap may come from better training data rather than from clue use.
  3. Clarification: Why are the base model's scores well below those in the Qwen3-VL technical report? Were all models in Table 2 evaluated with the same prompts and settings? I ask because the report lists 69.6 on MMMU and 69.1 on BLINK for this model, versus 53.3 and 59.9 in Table 2. Even the final model stays below the official base numbers.
  4. Clarification: What rule does the curriculum use?
Reviewer_DpVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Jigsaw reconstruction by training a VLM with RL algorithms. To achieve this, the author developed a two-stage pipeline: the first stage creates training data, and the second stage trains a VLM with RL. Specifically, in the first stage, the author fed an image into an advanced VLM (such as Qwen-30B) to generate possible clue descriptions for Jigsaw reconstruction. They then prompt the VLM again to reconstruct a shuffled image jigsaw using the clues. False reconstructions and false clues (judged by another VLM judge) are removed, keeping only correct reconstructions and reasoning clues for second-stage training. In the second-stage training, they first use SFT to tune a small VLM (Qwen-8B) to learn jigsaw reconstruction based on the training data obtained in the first stage. Additionally, they develop 4 types of subtasks, such as assuming the relation between pairwise patches, state judgment, Swap repair, and Candidate selection. They then use the DAPO RL algorithm to further tune the model. Experiments show that on the 2x2 and 3x3 jigsaw reconstruction tasks, the method surpasses existing large VLMs without specific tuning (Gemini 3.5 Flash, Qwen3-VL-235B)

Critical strengths
  1. The authors propose a two-stage pipeline to train a VLM to conduct 2x2 and 3x3 jigsaw reconstruction tasks. To do this, they first generate training data via clue generation, then tune a VLM with RL algorithms.

  2. The tuned VLM reaches 90+% accuracy on the 2x2 and 3x3 jigsaw reconstruction tasks.

Critical weaknesses
  1. The 2x2, 3x3 jigsaw reconstruction task seems to achieve saturated performance (90+% for Gemini 3.5 Flash and the authors’ tuned model). How about more complex settings such as 10x10, 20x20, 50x50? Another concern: why use 4 types of subtasks pairwise patches, state judgment, Swap repair, and Candidate selection), and why are they necessary?

  2. The authors only compared with VLMs not specifically tuned for this task (Gemini 3.5 Flash, Qwen3-VL-235B); some of them are not expected to perform well at Jigsaw reconstruction. How do they compare with existing, specifically designed Jigsaw reconstruction algorithms?

  3. The RL method used here is DAPO. Is there a reason to use it? A more generic option is GRPO. The reason for selecting DAPO is not clear to me.

  4. The numbers in Table 2 and Figure 3 don't match; is there any difference in settings? Also, how do you generate training data for generic QA tasks such as RealWorldQA? Some benchmarks only have test data.

Questions for authors

My concerns are mainly mentioned in the weaknesses:

  1. How about large jigsaw, 2x2, 3x3 reaches 90+%, how about 10x10, 20x20, 50x50?

  2. Why are the 4 types of subtasks necessary?

  3. Why not compare with specifically designed jigsaw algorithms?

  4. Why the DAPO RL algorithm instead of the more generic GRPO?

  5. Details about transferring to more generic benchmark.

1 comment

Thank you again for your time and thoughtful feedback. We would appreciate hearing whether our clarifications have addressed your concerns, or if any issues still need further explanation. We remain available to answer any follow-up questions.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely appreciate your time and consideration of our manuscript. We would like to respectfully draw your attention to the current discussion status: Reviewers DpVq and CqFQ (both Score 2) have not yet provided follow-up feedback on our rebuttal, submitted on November 13, despite our follow-up messages on November 17 and 18. In contrast, Reviewer 8HWn explicitly maintained their positive score of 4 after our response. We kindly ask you to weigh the two initial weak-rejection ratings in light of the clarifications and additional evidence summarized below.

  1. [Key Clarification] General benchmark gains come from jigsaw training alone.
    Our additional SFT and RL stages use only jigsaw-related data, without mixing in general-purpose RL or VQA data. We perform no benchmark-specific fine-tuning on the six general visual benchmarks. Their improvements therefore reflect transfer from jigsaw post-training. (Responses to Reviewer DpVq, Q5, and Reviewer CqFQ, Q2.)

  2. [Key Clarification] The reconstruction task is not saturated.
    Reviewer DpVq’s concern assumes that both puzzle sizes achieve over 90% accuracy. Our reported accuracy is 97.4% on 2×2 puzzles but 64.1% on 3×3 puzzles, leaving substantial room for improvement. We additionally evaluated 4×4 and 5×5 puzzles to characterize performance on larger grids and the current scaling limitations. (Response to Reviewer DpVq, Q1.)

  3. Additional comparisons with jigsaw-specific post-training methods.
    We retrained Visual Jigsaw and Jigsaw-R1 using the same Qwen3-VL-8B-Instruct backbone and shared full-restoration training data. ClueJigsaw outperforms both retrained baselines on both puzzle sizes and all six general visual benchmarks. These results address the missing comparisons and support the reconstruction and transfer benefits of our framework. (Responses to Reviewer DpVq, Q3; Reviewer 8HWn, Q1; and Reviewer CqFQ, Q1.)

  4. Additional ablations of the four auxiliary subtasks.
    Removing each auxiliary subtask individually, while retaining full restoration and the remaining subtasks, reduces 3×3 accuracy by 9.2–15.4 percentage points. These experiments substantiate the contribution of each subtask to the complete framework. (Response to Reviewer DpVq, Q2.)

We respectfully ask that this updated evidence be considered in the final assessment, particularly for the concerns underlying the two weak-rejection ratings that have not received follow-up discussion. We are fully committed to incorporating all constructive feedback into the final camera-ready version. Thank you for your time and thoughtful consideration of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.