- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VOTEX uses multiple samples of the same compositional prompt to construct self-distillation targets. After a joint-attention block incorporates each sample’s visual information into its prompt-token states, the next block lets image queries read prompt-token banks from all samples. A normalization term preserves attention under duplicated banks. The resulting predictions supervise an ordinary single-sample model, retaining its inference architecture and cost. Evaluation covers three backbones, six benchmarks, component controls, and donor-intervention experiments.
Critical strengths
-
The cross-sample correspondence is appealing and technically concrete. Prompt-token positions offer a shared interface despite different image layouts. The normalized group read and dense distillation provide a plausible way to exploit complementary samples without retaining group computation during deployment.
-
The reported empirical evaluation is extensive. Improvements are reported across three backbones and six benchmarks, with repeated training runs, human evaluation, and several informative ablations. In particular, the visual-write and cross-prompt controls help test whether the proposed information path contributes to the gains.
Critical weaknesses
-
The group-size results contain unresolved reporting discrepancies. Table 3 reports scores of 51.4 and 58.3 at G=1 and G=4, whereas Figure 16 shows approximately 20.4 and 26.7 without identifying a different evaluation setting. The two G=4 budget tracks also report different results despite apparently identical training budgets, weakening the scaling analysis.
-
The helpful-donor intervention is insufficiently specified. A donor selected as helpful in one bank is identified by the same index in another bank with disjoint seeds. Under independent, exchangeable sampling, that index need not remain helpful. Without the reported role-repetition rates and selection details, the masking contrast does not clearly establish selective transfer of correct content.
Questions for authors
-
Please reconcile Table 3, Figure 16, and the G=4 budget results by specifying their datasets, metrics, checkpoints, and individual runs.
-
How are donor roles established in the intervention bank, and what are the role-repetition rates and uncertainty estimates for the masking contrast?
3 comments
Response to Reviewer qMZB (2/2): donor roles, repetition rates, and uncertainty
(2/2) Donor roles in the masking experiment
Roles are now assigned inside the bank that is read. Your argument is correct as stated: if the intervention bank were an independent, exchangeable redraw, a donor index chosen in another bank would carry no role, and the helpful and unrelated masks would have the same expected effect. We answer in two parts.
(a) What the submission's design did, with the repetition rates. The intervention bank kept each donor's initial latent and redrew only the target's; "disjoint generation seeds" referred to the target seeds, which the text should have said (App. B.4 now does). Because the donors are not redrawn, roles mostly persist, but not always:
| SD3.5 | FLUX | |
|---|---|---|
| attempts | 2214 | 2147 |
| helpful index correct again | 81.8% | 80.2% |
| unrelated index wrong again | 86.0% | 85.3% |
| donor accuracy on target-missed atoms (independent-redraw reference) | 44.1% | 44.7% |
So the old contrast (Table 6) mixes attempts in which the role held with attempts in which it did not, and some of its target atoms were no longer missed in the new bank, which is why its full-read accuracy (75.84%) is higher than the recoveries below. It is a diluted estimate, and we no longer use it as the main evidence.
(b) The in-bank design (new Table 3, Sec. 4.3). For each GenEval2 prompt the frozen teacher draws samples, and each serves once as target. An atom is missed when the target's local read, with the same noise, gets it wrong. Donor roles come from the donors' own images in the same group, so nothing is transferred. We keep cases with 1 or 2 of the 3 donors correct (1880 on SD3.5, 1832 on FLUX) and mask one donor's text bank at every step.
| Recovery of missed atoms (%) | SD3.5 | FLUX |
|---|---|---|
| full read | 53.7 | 55.0 |
| helpful donor masked | 24.6 | 26.1 |
| incorrect donor masked (matched on other atoms) | 49.7 | 51.3 |
| helpful bank replaced by another prompt's | 20.2 | 21.9 |
| contrast, pp [95% interval] | 25.1 [22.4, 27.8] | 25.2 [22.3, 28.1] |
Intervals resample prompt clusters 20,000 times, since the four targets of a prompt share donors. The helpful mask removes 7.3 and 7.8 times as much recovery as the incorrect-donor mask; averaging over the group would predict similar drops. Table 9 adds the attention side: with only one of three donors correct, the image queries of the missed atom place 0.448 of their relevant text mass on it against 0.276 on each incorrect donor (SD3.5).
Changes in the paper: Sec. 4.3 rewritten around Table 3; App. B.4 corrected; App. C.5 with the repetition rates and Tables 9 and 10; the abstract now says "at least four times", which holds for both designs.
For the AC discussion, in brief. (i) The mechanism you found appealing now has a direct test at each step: copies or other prompts remove at least 90% of the gain (Table 5), masking the specific donor that rendered a missed requirement removes most of its recovery (Table 3), and attention favors that donor even when outnumbered (Table 9). (ii) Every group-size run is listed under one protocol (Table 8). (iii) With the strongest baselines on all three backbones, VoTEx stays best on GenEval2, T2I-CompBench++, and DPG, and on GenEval among held-out methods, without a reward model (Table 1).
Your questions led to the two clearest new pieces of evidence in the revision. If they now read as resolved, we would be grateful if you would consider raising your score and supporting the paper in the discussion.
Thank you for the rebuttal. My concerns have been addressed, and I will maintain my original rating.
Response to Reviewer qMZB (1/2): group-size numbers reconciled
(1/2) Group-size results
Thank you for a review that is both kind and exacting. Your summary states the method more crisply than our own introduction (prompt-token positions as the shared interface, the normalized group read, dense distillation with unchanged deployment), and you identified exactly the two places where a reader could not verify what we wrote. We took both seriously; this post answers W1/Q1 and the next answers W2/Q2.
The three sets of numbers measure different things, and one budget was misstated. You are right that the submission did not let a reader reconcile them. Here is what each one is.
The two G=4 tracks did not share a budget. As App. B.1 described them they would have, and your reading was the correct one. The description was wrong: the fixed-trajectory track holds 4,800 trajectories per run, the budget of G=2 at 300 updates, so G=4 gets 150 updates and G=8 gets 75. Figure 16(c) already shows this design, since the two tracks share their G=2 point and separate only at G=4 and G=8. We corrected App. B.1 and rewrote the paragraph in Sec. 4.4.
Every run under one protocol (new Table 8, App. C.4; GenEval2 test GM, validation-selected checkpoints):
The fixed-trajectory G=2 entry is the fixed-update G=2 configuration (same three runs). The G=4 fixed-update runs are the main SD3.5 runs, and their gains over the base score of 51.34 reproduce the interval [6.24, 7.70] reported in Sec. 4.2. At G=1 the loss is identically zero, so the parameters never move; those runs differ from 51.34 only through non-deterministic attention kernels at generation time.
What the corrected numbers say. At an equal budget of 4,800 trajectories, G=4 still beats G=2 (57.1 against 56.0), so the choice of G=4 is not a side effect of extra rollouts. G=8 falls to 55.1 because the same 4,800 trajectories then cover 600 prompts in 75 updates, against 2,400 prompts in 300 updates at G=2: a larger group raises coverage per prompt (Theorem 1), and at a fixed budget it pays for that with prompts and updates. Validation-selected checkpoints score at or above the final checkpoints in every case, so the earlier sentence understated the fixed-trajectory track rather than overstated it.
Changes in the paper: Sec. 4.4 (rewritten paragraph), App. B.1 (budget), Figure 16 caption (prompts and score), App. C.4 with Table 8.