- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines cases where VLMs miss a word or object outline embedded in a ControlNet-generated scene. SemVink and SMSP try to expose the hidden pattern through downsampling and low-pass views, respectively. ControlTrace trains an 8.4M-parameter U-Net using carrier images as inputs and their original grayscale control fields as reconstruction targets. For a new image, the network predicts the field without needing the original generation inputs. Its prediction becomes the visual input to a VLM, which identifies the hidden content. The comparison uses FreqBlind, where 6,000 images are divided equally between contour and text targets. Both groups cover three conditioning strengths, with real words and pseudowords making up the text group. Using Qwen2.5-VL-7B, ControlTrace recognizes 60.2% of the contours, compared with 33.3% for SMSP. Their contour scores are furthest apart at the weakest setting.
Critical strengths
The paper includes extensive experiments. The real-word, anagram, and pseudoword tests examine the reader’s reliance on familiar words. Natural and synthetic images with no embedded target show how often the reader reports one anyway. Hidden-content recognition has a practical role in moderation, since the classifiers and VLMs tested on Hateful Illusion often missed hidden hateful messages.
ControlTrace takes just 7.4 ms more per image (5.3%) than direct reading in the reported tests. The tested AVR implementation needs seven VLM calls for each image; ControlTrace needs only one to read the recovered field.
Critical weaknesses
The authors train a U-Net to recover control fields from the corresponding generated images, using standard reconstruction losses. Without the edge head, weak-contour accuracy is just 0.27 percentage points lower. The deeper versions they tested do no better. NAFNet reaches 56.51% overall contour accuracy when trained to recover the same control fields, leaving a 3.70-point gap to ControlTrace. Although the supervision-target ablation supports choosing control fields, the improvement beyond a basic supervised reconstruction model needs to be better established.
With SD1.5/QR Code, SMSP gets 22.56% contour accuracy; ControlTrace manages only 4.78%. This is a concern for moderation if an attacker can choose or fine-tune the control encoder to exploit that sensitivity. I would have liked to see the authors try training on several generators or adapting the model to a new encoder.
Questions for authors
Why is ControlTrace absent from the HC-Bench evaluation? Can the authors report its results under the same evaluation protocol as the baselines?
With the same training data and comparable tuning budgets, how much does the full method improve over a U-Net trained to recover control fields using only L1 and cosine losses, and over a comparably tuned NAFNet? Do the main conclusions still hold when the gradient loss and auxiliary edge head are removed together?
3 comments
Thanks for running these extra experiments. It was good to see that the simpler U-Net does about as well on FreqBlind. That makes me more convinced that recovering the control field is a useful way to pick up hidden contours. I’m glad you also ran the HC-Bench comparison, since it shows how much harder things can get on a different dataset.
That’s the one point I’d bring out a bit more in the main text. The frozen model gets 4.17% on HC-Bench objects, and that goes up to 23.21% after fine-tuning with separate photo-based controls. Putting those numbers side by side would help readers see where the original model struggles and how much fine-tuning helped here. Overall, I’m more confident in the main finding after reading the response, and I still lean toward acceptance.
Response to Reviewer 1Qhh (follow-up): thank you
Thank you for reading our response and the new experiments so carefully, and for your continued support.
Your comments have enriched the paper and made it clearer. They led us to compare against a basic U-Net and a comparably tuned NAFNet, evaluate ControlTrace on HC-Bench and fine-tune it with photo controls, and train on a second encoder; all of these results are now in the revised PDF (Appendices B.2, C.2 and E.3). Together, they show where the contour gain comes from and where the frozen model needs further training.
Response to Reviewer 1Qhh: loss and backbone ablations, HC-Bench, and encoder training
Thank you for the close reading of our ablations. Your questions led us to test what the contour gain depends on and to evaluate ControlTrace on HC-Bench; all new results are now in the revised PDF, with changes marked in blue.
Q2 / W1. Basic U-Net, removing both terms, and a comparably tuned NAFNet. The main conclusions hold without the gradient loss and the edge head. All rows except SMSP use the same four-level RGB U-Net, data, learning rate and 30 epochs; accuracy (%) is the mean of three seeds, and intervals pair the 50 contour targets.
Appendix C.2 (Table R4) reports this ablation, and Section 5.3 now notes that removing both terms leaves accuracy unchanged. Five- and six-level U-Nets did not improve weak contours.
For the backbone comparison (Appendix C.2, Table R5), both networks used L1 + cosine, the same data and 30 epochs; at seed 42 each tried three learning rates, and validation MAE selected 0.0001 (U-Net) and 0.0002 (NAFNet) for all seeds.
Both backbones far exceed SMSP, U-Net leads under equal tuning, and the overall gap is similar to the paper's (4.41 vs 3.70 pp). Together with Table 12, where the submitted U-Net reaches only 13.27% on weak contours when trained to output the s = 2.0 carrier instead of the field, this places the gain in the recovery target rather than in the losses or the backbone.
Q1. HC-Bench. Table 8 was meant only to check our baseline implementations. We have now evaluated the three frozen checkpoints on the same 84 images (56 objects, 28 text), reader, prompts and scorer. Accuracy (%):
ControlTrace fails on HC-Bench objects: they come from photographs (continuous tones, light or dark foregrounds), while we trained only on white-on-black icons. In a test designed after seeing these failures, we fine-tuned ControlTrace without HC-Bench images (Table R2). Object accuracy (%, HC-Bench prompt):
The photo controls add +18.45 pp [8.93, 29.17] over HC-Bench-like settings alone. Table 8 and Table R2 (Appendix B.2) now include these rows.
W2. Training on several encoders. The SD1.5/QR Code failure is in recovery, not reading (foreground MAE 0.121 with QR Code Monster vs 0.957 with QR Code, reader fixed). We mixed QR Code Monster and QR Code training pairs (9,600 each) and trained 10 epochs with three seeds, generating both encoders' training carriers at the strength s given in each row. Tests use the paper's SD1.5 sets (900 carriers each; weak = the 300 at s = 1.0). Accuracy (%; Appendix E.3, Table R9):
Adding the second encoder helps at medium and strong conditioning (from scratch at s = 2.0: QR Code, all strengths, 31.74% vs SMSP 22.56%) but not on the weakest images, where no method exceeds about 5%. Training at stronger s also lowers weak QR Code Monster accuracy (34.67% → 13.67%) and QR Pattern accuracy (35.48% → 30.48%). A known encoder can thus be covered at some cost; robustness to an encoder chosen to defeat the recovery model remains open, as Section 6 now states.