- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Pentimento, a conditional diffusion model for recovering brushstroke sequences from finished paintings. The method uses a convolutional encoder to extract image features and a Transformer-based diffusion model to generate ordered stroke sequences. The authors also introduce SynBrush, a synthetic dataset containing paintings with known stroke orders. Experiments investigate the effects of different painting priors, architectural designs, and conditioning strategies.
Critical strengths
- The paper explores a relatively underexplored research problem of recovering the original brushstroke order from finished paintings.
- The authors introduce a new synthetic dataset, SynBrush, with ground-truth stroke-order annotations, enabling quantitative evaluation of the proposed method.
Critical weaknesses
-
The proposed Pentimento framework mainly adapts existing diffusion-based action sequence models, such as Diffusion Policy and Diffuser, to brushstroke-order recovery. The core architecture consists of a convolutional image encoder and a Transformer-based diffusion denoiser, making the methodological novelty somewhat limited. Although the authors introduce region attention and curriculum learning in Section 3.4, the experimental results do not consistently support their effectiveness. As shown in Table 4, the baseline achieves a mean Kendall's of ; adding region attention improves it only to , while combining region attention with curriculum learning reduces it to . In contrast, simply increasing the training steps from to improves the mean τ to . These results raise questions about whether the proposed architectural modifications provide meaningful benefits beyond existing diffusion-based sequence modeling techniques. I would appreciate further clarification of the specific methodological contributions and stronger experimental evidence supporting the effectiveness of the proposed improvements.
-
In Section 4, the authors construct SynBrush, a synthetic dataset generated using predefined rules for stroke widths, colors, layers, and spatial regions. However, as Figure 3 shows, the generated paintings are relatively simple and differ substantially from real-world artworks in visual complexity, painting style, and stroke composition. Although the synthetic dataset provides accurate ground-truth stroke orders, it remains unclear whether the proposed method can generalize to realistic painting scenarios. Moreover, existing studies have explored painting process reconstruction using real artwork and painting sequence data. Therefore, I would like the authors to better justify using such simplified synthetic paintings and, if possible, provide additional experiments on real-world painting data to demonstrate the practical applicability and generalizability of the proposed method.
Questions for authors
- What evidence supports the benefits of region attention and curriculum learning over the baseline under matched training budgets?
- How well can Pentimento generalize from SynBrush to real-world paintings with recorded stroke histories?
- Is the observed prior dominance mainly caused by SynBrush's predefined ordering rules or limitations of the conditioning architecture?
2 comments
One correction to our answer to your Q3. We said the bottleneck was the architecture, but at that point we hadn't actually tested a different one. We tried higher-resolution encoders (32x32 and 64x64) and a larger denoiser: they still don't make Pentimento use the image. A model given the true strokes ignores the image too when it reads it the same way (tau 0.374 vs. 0.379 with no image), but uses it once it reads the image along each stroke (0.433). So our answer stands, now with evidence behind it: the bottleneck is how the model reads the image, not the rules that generate the data.
Details are added in Section 6 and Appendix J of the revised PDF. Thank you again for your careful review!
Thank you for the careful reading and the direct questions.
W1 / Q1 (method novelty; evidence for region attention and the curriculum). We agree. At the same training budget, neither helps beyond seed noise: tau 0.286 +/- 0.017 (region attention), 0.252 +/- 0.029 (curriculum) and 0.260 +/- 0.020 (both), against 0.270 +/- 0.006 for the baseline. We no longer present them as contributions and moved them to Appendix C. The contributions are: the unpainting task, posed as inferring a distribution over stroke orders; two new components for it, SynBrush (a generator and benchmark with exact stroke order) and Pentimento (a conditional diffusion model that samples stroke sequences); and the finding that recovered order comes mostly from a learned prior. We built Pentimento on a standard model class on purpose, so the finding applies to that class rather than to one custom design.
W2 / Q2 (synthetic data; generalization to real paintings). We cannot measure this yet, because no public real-painting dataset we know of records stroke-level order. Painting videos, as used by Inverse Painting and Loomis Painter, record pixels, not strokes, and give only frame-level order (Sec. 2, new App. D). Logged digital paintings, which record every stroke, are the real-data next step (Sec. 6). Synthetic data is a deliberate choice for a diagnosis that needs exact order.
Q3 (SynBrush rules or the conditioning architecture?). We separated the two. -- Rules: when stroke width carries no order, accuracy falls from 0.27 to 0.11, but the model still ignores the image, even though the image still determines much of the order (exact occlusion sort: 0.41). -- Architecture: applied to the model's own predicted strokes, the occlusion rule scores -0.08, because the strokes are placed too inaccurately to read which is on top. Our answer is the architecture, not the rules: the model neither places strokes precisely enough to use occlusion nor takes order from the image, consistent with the coarse 16 x 16 image encoding discussed in Sec. 6.