acceptodds
Under review as a conference paper at ICLR 2027

Unpainting: Characterizing Brushstroke Order Recovery from Finished Paintings

Abstract

When a painting is finished, the order in which its strokes were laid down is no longer recorded; only the final surface survives. We ask how much of that hidden order a painting can still reveal. We call this unpainting: recovering the stroke sequence behind a finished canvas. We introduce Pentimento, a conditional diffusion model, and SynBrush, the synthetic corpus of known-order paintings we train it on; rather than guessing one answer, it proposes many plausible orderings and reports how much they agree, with pairwise probabilities well calibrated on held-out paintings. Recovery is real but limited: far better than chance, yet well short of certainty. Targeted interventions trace it predominantly to the learned process prior: hand the model a stranger's canvas, or none at all, and it does just as well; a model trained without a canvas reaches Kendall's against the conditional model's . Meanwhile, a simple rule asking only which stroke covers which recovers twice as much of the order, though it is handed the strokes the model must find. Conditioning closes 5% of that gap: it says substantially more about the stroke parameters than about their order. The finished canvas remembers more about how it was made than the model takes from it. What Pentimento does offer is calibrated confidence in its own samples: answer only the fifth it is surest about and it is right 97% of the time. That confidence is prior-driven too, barely better than a stroke-width heuristic, uncorrelated with what a given canvas determines. We offer unpainting as a controlled study of prior dominance in amortized inverse inference, not a tool for art historians.

Primary area:
generative models
Keywords:
diffusion models, inverse problems, amortized inference, uncertainty quantification, calibration, selective prediction, sequence generation, synthetic benchmark, computational art history

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fDpyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Pentimento, a conditional diffusion model for recovering brushstroke sequences from finished paintings. The method uses a convolutional encoder to extract image features and a Transformer-based diffusion model to generate ordered stroke sequences. The authors also introduce SynBrush, a synthetic dataset containing paintings with known stroke orders. Experiments investigate the effects of different painting priors, architectural designs, and conditioning strategies.

Critical strengths
  1. The paper explores a relatively underexplored research problem of recovering the original brushstroke order from finished paintings.
  2. The authors introduce a new synthetic dataset, SynBrush, with ground-truth stroke-order annotations, enabling quantitative evaluation of the proposed method.
Critical weaknesses
  1. The proposed Pentimento framework mainly adapts existing diffusion-based action sequence models, such as Diffusion Policy and Diffuser, to brushstroke-order recovery. The core architecture consists of a convolutional image encoder and a Transformer-based diffusion denoiser, making the methodological novelty somewhat limited. Although the authors introduce region attention and curriculum learning in Section 3.4, the experimental results do not consistently support their effectiveness. As shown in Table 4, the baseline achieves a mean Kendall's τ\tau of 0.2700.270; adding region attention improves it only to 0.2860.286, while combining region attention with curriculum learning reduces it to 0.2600.260. In contrast, simply increasing the training steps from 5,0005,000 to 20,00020,000 improves the mean τ to 0.3430.343. These results raise questions about whether the proposed architectural modifications provide meaningful benefits beyond existing diffusion-based sequence modeling techniques. I would appreciate further clarification of the specific methodological contributions and stronger experimental evidence supporting the effectiveness of the proposed improvements.

  2. In Section 4, the authors construct SynBrush, a synthetic dataset generated using predefined rules for stroke widths, colors, layers, and spatial regions. However, as Figure 3 shows, the generated paintings are relatively simple and differ substantially from real-world artworks in visual complexity, painting style, and stroke composition. Although the synthetic dataset provides accurate ground-truth stroke orders, it remains unclear whether the proposed method can generalize to realistic painting scenarios. Moreover, existing studies have explored painting process reconstruction using real artwork and painting sequence data. Therefore, I would like the authors to better justify using such simplified synthetic paintings and, if possible, provide additional experiments on real-world painting data to demonstrate the practical applicability and generalizability of the proposed method.

Questions for authors
  1. What evidence supports the benefits of region attention and curriculum learning over the baseline under matched training budgets?
  2. How well can Pentimento generalize from SynBrush to real-world paintings with recorded stroke histories?
  3. Is the observed prior dominance mainly caused by SynBrush's predefined ordering rules or limitations of the conditioning architecture?
2 comments

Thank you for the careful reading and the direct questions.

W1 / Q1 (method novelty; evidence for region attention and the curriculum). We agree. At the same training budget, neither helps beyond seed noise: tau 0.286 +/- 0.017 (region attention), 0.252 +/- 0.029 (curriculum) and 0.260 +/- 0.020 (both), against 0.270 +/- 0.006 for the baseline. We no longer present them as contributions and moved them to Appendix C. The contributions are: the unpainting task, posed as inferring a distribution over stroke orders; two new components for it, SynBrush (a generator and benchmark with exact stroke order) and Pentimento (a conditional diffusion model that samples stroke sequences); and the finding that recovered order comes mostly from a learned prior. We built Pentimento on a standard model class on purpose, so the finding applies to that class rather than to one custom design.

W2 / Q2 (synthetic data; generalization to real paintings). We cannot measure this yet, because no public real-painting dataset we know of records stroke-level order. Painting videos, as used by Inverse Painting and Loomis Painter, record pixels, not strokes, and give only frame-level order (Sec. 2, new App. D). Logged digital paintings, which record every stroke, are the real-data next step (Sec. 6). Synthetic data is a deliberate choice for a diagnosis that needs exact order.

Q3 (SynBrush rules or the conditioning architecture?). We separated the two. -- Rules: when stroke width carries no order, accuracy falls from 0.27 to 0.11, but the model still ignores the image, even though the image still determines much of the order (exact occlusion sort: 0.41). -- Architecture: applied to the model's own predicted strokes, the occlusion rule scores -0.08, because the strokes are placed too inaccurately to read which is on top. Our answer is the architecture, not the rules: the model neither places strokes precisely enough to use occlusion nor takes order from the image, consistent with the coarse 16 x 16 image encoding discussed in Sec. 6.

One correction to our answer to your Q3. We said the bottleneck was the architecture, but at that point we hadn't actually tested a different one. We tried higher-resolution encoders (32x32 and 64x64) and a larger denoiser: they still don't make Pentimento use the image. A model given the true strokes ignores the image too when it reads it the same way (tau 0.374 vs. 0.379 with no image), but uses it once it reads the image along each stroke (0.433). So our answer stands, now with evidence behind it: the bottleneck is how the model reads the image, not the rules that generate the data.

Details are added in Section 6 and Appendix J of the revised PDF. Thank you again for your careful review!

Reviewer_zBSyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies brushstroke-order recovery from finished paintings and finds that its model relies mainly on learned painting patterns rather than canvas-specific evidence.

Critical strengths

Recovering brushstroke order offers an interesting perspective on inferring how an image was created.

Conditioning interventions and unconditional training provide useful evidence about the model’s limited reliance on the canvas.

Critical weaknesses

The main experiments use only 90 training paintings per setting, and the larger dataset follows similar generation rules, leaving open whether the findings primarily reflect a small, narrowly patterned training distribution.

The occlusion-based reference receives the true stroke set, so its advantage does not directly quantify how much image evidence the model fails to extract.

Questions for authors

Beyond randomizing stroke parameters, how much do the underlying generation patterns vary, particularly the coarse-to-fine progression and layer-dependent color rules?

Does the limited benefit of canvas conditioning persist with larger training sets covering substantially different painting-order patterns?

Can canvas evidence be evaluated under the same input conditions, without giving the reference method the true stroke set?

3 comments

Thank you for a clear and fair review. We agree with both weaknesses and ran an experiment for each.

W1 / Q2 (does the weak use of the image persist with more and different data?). Two new experiments. More data: with up to 5,000 paintings, tau stays at about 0.27 (0.35 with 90). The image becomes slightly useful (+0.04), but the dataset's average image still beats each painting's own image at every size (Table 4). Different rules: we built a SynBrush variant in which stroke width carries no order. The model reaches tau = 0.11, and a model trained without the image reaches 0.10 (85% of it). Another painting's image changes nothing, and a blank image even scores slightly higher than the real one, on every seed. Yet the image still determines much of the order (exact occlusion sort: 0.41). So the finding persists when the main cue is removed (Sec. 5.4, App. H).

W2 / Q3 (the occlusion reference saw the true strokes; compare under the same inputs). Considering this criticism, we now make the comparison fair. We apply the occlusion rule to the model's own predicted strokes, reading which stroke is on top from the real image only, so it sees exactly what the model sees. It scores tau = -0.08, against 0.29 for the model, because predicted strokes overlap the true ones by only 10%. Under equal inputs the occlusion rule does not beat the model, and the earlier gap is mostly in recovering the strokes. Sec. 5.3 is rewritten accordingly. Thank you for pushing on this.

Q1 (how much do the generation patterns vary?). Honestly, not much beyond the random stroke settings. All levels paint coarse to fine over 2 to 4 layers, with color by layer and regions as optional additions. So a width cue is always present, and it is strong: sorting the true strokes by width alone gives tau = 0.55. The paper now says this plainly (Sec. 4), and the width-free variant above tests what happens without it.

Thank you for the author's response, but simply increasing the quantity using the same rules and variations is not sufficient to demonstrate applicability to a broader training distribution. I still maintain that training on generative lines is the fundamental issue; while this simplifies the data source for the task, it also limits its persuasiveness. It is recommended to conduct experiments on real artworks of different styles, which would be more convincing.

Thank you for the follow-up! We agree that more data made with the same rules can't settle this, so we tried your suggestion and ran the model on real artworks. We picked 10 paintings in very different styles: Leonardo, Vermeer, Rembrandt, Monet, Seurat, Van Gogh, Munch, Klimt, Kandinsky and Mondrian. Real paintings don't come with a record of their stroke order, so we can't score accuracy on them. What we can check is whether the order the model produces depends on the painting at all.

Keeping the random seed fixed, we compared the orders it produced (Kendall tau, where 1 means the same order):

  • given a different painting instead: 0.99
  • given a blank image instead: 1.00
  • same painting, different random seed: 0.22 So for all ten styles, the painting itself barely changes the order, just as on our synthetic paintings (0.98, 0.99 and 0.24). The finding isn't something special to synthetic images.

We want to be upfront about two limits, which the new App. K states plainly. The model's strokes don't look like the real paintings, so it doesn't carry over to real art, as you expected. And we can't tell whether an order recovered from a real painting is correct until there is data that records real strokes, such as logged digital paintings (Sec. 6).

Beyond adding data, we also changed the rules: in the width-free variant, stroke width says nothing about order, and the finding still holds. A new experiment explains why: Pentimento's way of reading the image doesn't pick up the order, while a model that reads the image along each stroke does (Appendix J).

Thank you for pushing us toward testing real art! It made the paper more honest about what it does and doesn't show.

Reviewer_xvvzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Unpainting, where given a canvas or a finished painting, the task is to recover the order in which the brushstrokes were actually done (or can be done) to create that particular painting. Since there can be multiple such orders to create the same painting, the authors treat this task as sampling from a distribution over the stroke orders rather than simply just predicting one "right" answer. They also release SynBrush, a procedural dataset of paintings with known stroke orders.

The paper trains a conditional diffusion model called Pentimento that generates a sequence of parametric strokes conditioned on the finished painting. During inference, they sample many orderings per painting and use their spread as a measure of confidence.

The main claims in the paper are:

  1. The stroke order is partially recoverable with Pentimento reaching a Kendall tau of about 0.30, where 0 is random and 1 is the exact order.

  2. Most of the model's recovery comes from the learned priors over all training data, rather than the actual condition painting itself.

  3. The model's confidence is very well calibrated, but it might also be coming mostly from the prior, since a simple stroke width rule gives almost the same confidence ranking.

Critical strengths

The novel task of unpainting is quite interesting. Thinking about a painting as a process rather than a final image or a product is a very artistic framing. Also, treating the answer as a distribution instead of a single right order, I think, is the right way to handle the ambiguity and at the same time open avenues for diffusion models to enter the framing.

Since the authors introduce this new task, also releasing a dataset is a very good choice.

Critical weaknesses
  1. Since the main training runs only use 90 paintings on a small 4-layer model, I wonder if this is the reason why the recovery comes more from the prior and not the canvas. I would encourage the authors to try much more data with longer training, a stronger image encoder, maybe a bigger model too.

  2. There seems to be prior work on recovering stroke orders from static images in line-drawing or handwriting setups. The paper should cite these lines of work and better position their work amongst these. (Kato & Yasuhara (TPAMI 2000), Fu et al. (SIGGRAPH Asia 2011) and TRACE (ICDAR 2021))

  3. All the experimental results are from the synthetic dataset. Since the paper is motivated by art history and prior works (e.g. Inverse Painting, Chen et al., SIGGRAPH Asia 2024, and Loomis Painter, Pobitzer et al., 2025) seem to be evaluating on real painting time-lapses, a small real data evaluation would make the paper much stronger.

  4. The abstract is hard to follow (e.g. "The finished canvas remembers more about how it was made than the model takes from it").

  5. Sevaral figures are not polished and have a lot of overlapping elements and text.

  6. Since the model mostly seems to be ignoring the conditioning, a natural baseline would be a diffusion sampler that checks its outputs against a condition e.g. diffusion posterior sampling with a differentiable stroke renderer. Another experiment would be to run the occlusion sort on the model's own predicted strokes. This would show whether the gap comes from recovering the strokes or from recovering their order.

Questions for authors

How does the tau value change with scaling training images or paintings? for example tau on 1k vs 5k vs10k images?

How does the occlusion sort work for Pentimento's predicted stroke set instead of the actual true set?

Have you tried any guided sampling for inference?

Please also refer to the weaknesses for more thorough questions

4 comments

Thank you for the detailed review and the concrete experiment suggestions. We ran them, and they sharpened the paper.

W1 / Q1 (training set size; tau at larger scales). We trained on 90, 250, 500, 1,000 and 5,000 paintings (20k to 40k training steps) and tested on the same 100 held-out paintings. Tau was 0.348, 0.295, 0.266, 0.265 and 0.268. Tau does not improve with more data, and it is flat from 1,000 to 5,000, so we did not continue to 10,000 within the discussion period. The model does start to use the image a little: from 1,000 paintings, training with the image beats training without it by 0.04 (95% CI 0.02 to 0.05). But at every size, giving the model the dataset's average image instead of the painting's own image scores higher, and 0.04 is small next to the 0.58 that the occlusion rule reaches on the true strokes. One caveat, stated in the paper: training steps grew less than data. (New Sec. 5.4, Table 4.)

W2 (missing handwriting and line-drawing work). Thank you for these pointers. We now cite and discuss Kato and Yasuhara (TPAMI 2000), Fu et al. (SIGGRAPH Asia 2011) and TRACE (ICDAR 2021) in Sec. 2. These recover the order of thin strokes from continuity at junctions and pen movement. Painted strokes are wide, colored and often fully covered (16.8% of SynBrush strokes leave no visible trace), so the image determines much less of the order. This is why we model a distribution over orders.

W3 (real paintings, as in Inverse Painting and Loomis Painter). We agree these works use real painting processes, so we checked what their data can measure. Both train and evaluate on recorded painting videos and compare frames with image metrics. Order is judged only at frame or region level or by experts, and neither dataset is public. The core limit is that a video records pixels, not strokes: each frame difference merges many strokes, and splitting it into ordered strokes is exactly our problem, with no ground truth to check against. A new appendix (App. D) shows this on SynBrush with a perfect simulated video. We keep the synthetic setting on purpose: if a model does not use occlusion evidence where it is clean and exactly measurable, real paintings are unlikely to change that. Logged digital paintings, which record every stroke, are the real-data next step (Sec. 6).

W4 (abstract hard to follow). Agreed. The abstract is rewritten in a cleaner flow, and the sentence you quoted is gone.

W5 (overlapping figure labels). Thank you. Figs. 1, 2, 5 and 6 are redrawn so that legends no longer overlap and axes end at the last tick.

W6 / Q3 (guided sampling, i.e. using the image at inference). This is a good suggestion that we could not run in the discussion period: guided sampling corrects each step with the gradient of a rendering loss, and our renderer is not differentiable. We now name it as the natural next baseline in Sec. 6.

Q2 (occlusion sort on the model's predicted strokes). Thank you; this was the most informative experiment. For each held-out painting we took the model's predicted strokes, decided which of each overlapping pair is on top by reading the real image (no ground truth), and ranked them. The model's own order scores tau = 0.29. The occlusion order on its predicted strokes scores -0.08, and combining the two only hurts (0.16). On the true strokes, the same rule reaches 0.42 (0.58 with exact occlusion). The reason is that predicted strokes overlap their true counterparts by only 10%, so occlusion cannot be read where they are. The gap you asked us to split is mostly in recovering the strokes, and only then in ordering them. Sec. 5.3 now says this, with details in App. F.

I appreciate the authors for running additional experiments. Some of my main concerns still remain open:

  1. When responding to reviewer fDpy, the authors claim that the bottleneck is the architecture and not the generation rules. This is exactly the experiment I asked in W1, but it was not run: all results still use the same encoder and model size. Without at least one or two additional higher-resolution encoders, I don't think the finding can be stated for this model class in general.

  2. It does look like the conditioning path in pentimento is not working. In the new width-free variant, where stroke width no longer carries any order information, occlusion sort on the true strokes still reaches a Kendall tau of 0.41. So the image itself still determines a good part of the order. Yet Pentimento only reaches 0.11 on the same variant.

  3. If I understand correctly, the predicted strokes overlap the true ones by only 10%. And the tau is computed after matching them, so I am not sure how much of tau = 0.29 reflects recovering this painting's order versus the generic coarse-to-fine prior. I wonder if a model that is given the true stroke set together with the canvas, and only predicts their order, would separate stroke recovery from the order recovery cleanly.

My main concern is that while the current evidence shows that Pentimento barely uses the image to recover the order, it still remains open whether the prior dominance is a property of the task itself or simply a failure of this specific model's conditioning path. The revised paper leaves this as an open question, but I believe this point needs to be addressed before making the central claim in the paper. I encourage the authors to push back on my point.

Thank you for coming back to this, and for being so specific. You were right on all three points, and the experiment you suggested in point 3 ended up being very helpful as described below:

  1. Higher-resolution encoders. We tried 32x32 and 64x64 encoders, and also a denoiser twice as large, with 3 seeds each on both generators. In all 21 runs, Pentimento still doesn't use the image: swapping in another painting's image changes tau by at most 0.006, and never significantly.

2 and 3. Your order-only model. We built it as you described: it gets the true strokes and the image, and only has to put the strokes in order. On the width-free data with 1,000 paintings:

  • with no image, it reaches tau 0.379
  • with Pentimento's cross-attention, 0.374, so the image adds nothing
  • when it instead reads the image along each stroke, 0.433, which beats the occlusion sort (0.41). Given another painting's image, it drops back to 0.369.

So the order really is in the image, and a learned model can pick it up, just not through Pentimento's cross-attention. Pentimento also has a second hurdle: reading the image along a stroke only works if you know where the stroke is, and its predicted strokes overlap the true ones by only 10%.

We've updated the paper to say this. Sec. 6 used to leave this as an open question and now addresses it properly. The abstract and conclusion are re-worded, and the full results are in the new Appendix J (and Table 9).

Thanks again for pushing us on this. The paper is much clearer for it.

Thank you for running these experiments. The order-only version is the most interesting one according to me. It changes the framing of the paper in a more cleaner and interesting way: the order is present in the image and a learned model can read it, but not through pentimento's cross-attention.

Since this result now carries the weight of the main claim, I recommend the authors to move it into main paper and report the seeds and confidence intervals for the order-only comparison.

I am raising my score. Good work on the rebuttals!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.