- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the interaction between output serialisation and LoRA rank in VLMs, showing that losslessly splitting depth-map generation into four calls benefits low-rank adapters more than high-rank ones at the cost of additional image prefills, while no comparable interaction is observed for the tested compact segmentation masks.
Critical strengths
-
Clear research question and target-preserving experimental design. The paper examines whether output serialisation changes the benefit of increasing LoRA rank, rather than simply whether chunking improves performance. Its format-by-rank comparison preserves the complete quantised target and matches image exposure and optimiser updates. This distinguishes overall format gains from changes in the marginal value of rank, while ruling out improvements obtained by discarding target information.
-
Consistent empirical support for the main depth finding. The interaction is positive across all three training seeds, with per-seed paired-image confidence intervals above zero. A fit-free ordinal metric and parser-sensitivity checks further support the finding, reducing dependence on a single evaluation procedure. Beyond average fidelity, the paper also demonstrates a substantial reduction in low-fidelity complete predictions, showing that the improvement extends to the failure tail.
Critical weaknesses
-
The practical benefit of preserving the complete target remains unclear. Predicting a low-resolution depth map in a single call and then upsampling already achieves a score comparable to four-call generation (0.648 versus 0.642). Although the simpler approach loses detail, the paper should demonstrate whether preserving the complete target actually improves fine-detail prediction or downstream performance.
-
It is unclear whether parameter savings justify the additional computation. Low-rank chunked generation uses fewer adapter parameters but requires four image prefills. The reported timings are not controlled measurements under identical hardware conditions, leaving its practical advantages in latency, GPU memory usage, and throughput uncertain.
-
Task differences are confounded with output length and encoding. Depth outputs are long, whereas segmentation masks use a compact representation of only 64 hexadecimal characters. The stronger interaction for depth may therefore reflect differences in output representation rather than inherent task properties. Comparing different encodings within the same task would help clarify this distinction.
-
Capacity requirements have not been clearly distinguished from learning speed. The experiments compare only two ranks and train for half an epoch. Without longer training and intermediate-rank comparisons, it remains unclear whether chunking consistently reduces the required capacity or simply helps low-rank adapters learn faster under the current budget. Single-call generation might catch up with more training.
Questions for authors
Please address the questions and concerns raised in the Weaknesses section.
2 comments
W4 — required capacity versus learning speed
The direct test is the same recipe trained twice as long: 3,750 updates, a full epoch, all four cells, seed 0. On the registered 1,000 images the full-epoch interaction is , no detectable difference on the primary metric, with a positive ordinal-accuracy companion (, interval excluding zero); from update 1,875 to 3,750 of the same trajectory it changes by , with an interval excluding zero. Update 1,875 of that trajectory is half annealed; with the cosine completed at 1,875 the interaction is , with an interval excluding zero (reply to Hmn1, W2/Q2). Rank 64 adds the intermediate point on the capacity axis, and it places the interaction in the lower segment: it is already present between ranks 16 and 64, where the ranks are close. We report both rank segments and both budget points.
W1 — what preserving the complete target buys
The 0.648-versus-0.642 comparison is a tie on a score that cannot see within-block structure, because both numbers average over cells whose teacher values the coarse arm represents by a single block mean. Measuring the fine detail directly, on the frozen 1,000-image set and all three seeds:
Within-block ordering accuracy is the fraction of within-block cell pairs whose predicted order matches the teacher's, over pairs whose teacher values differ. Residual amplitude is the standard deviation of H(x)=x−upsample(2×2 block mean of x). Skill is $1-\lVert H(\text{pred})-H(\text{ref})\rVert / \lVert H(\text{ref})\rVert$, so zero means no better than predicting no detail at all. The last column rescores each arm's own prediction with its high-frequency residual removed.
Reading these two ways, as we think they should be read:
We also ran the downstream check on the full BLINK and CV-Bench sets. It holds the question, the images and the answering model fixed and varies only which readout supplies the depth descriptor, with every descriptor rendered by the same routine, so a difference between arms is a difference between readouts. Between supplying the complete target in four grouped calls and supplying it in one, answering through the frozen backbone, we see no detectable difference on either benchmark: +0.006 on BLINK and +0.000 on CV-Bench, and against the coarse one-call readout +0.012 and +0.001. The grouped configuration reaches those numbers with a sixteenth of the adapter parameters. Detaching the skill restores the frozen backbone's own answer on every one of the 3,160 questions, so what the skill adds while worn is also exactly what it takes away when removed.
W2 — do the parameter savings justify the extra computation
This is the controlled benchmark in our reply to Hmn1 (Q3), and it is now measured: same GPU, one process, one complete map as the request unit, the three arms interleaved, concurrency 1 and 8, with p50 and p95 latency, throughput and peak memory. The controlled benchmark replaces the earlier timings, which were amortised throughput at a fixed number of calls in flight, measured on different hosts. Peak memory stays in one class (19.0 to 23.5 GiB), and the grouped rank-16 configuration uses a sixteenth of the adapter parameters.
W3 — task differences versus output length and encoding
Expand12is the within-task control for exactly this: same task, same 144 independent values, 1,727 tokens instead of 431. It shows the length axis alone does not account for the depth gain. The complementary test, the same task under a different encoding, is a segmentation replication with a verbose per-cell encoding; at the measured 19.4-23.7 s per optimiser update that is four cells of ten to twelve hours each plus its own registration, so we name it as the next experiment. From the compact-mask result we claim a boundary: at 16x16 packed masks the interaction stays inside the predefined ±0.02 region while rank still helps, which bounds where the depth pattern holds.