- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets at mask-free one-shot personalized segmentation. The input is an unmasked RGB reference image of an unseen subject and its category name. The goal is to segment that specific instance in query images or video while rejecting same-category look-alikes.
The proposed PerSeg includes three stages:
- PerSeg-DPT: it tunes per-identity prompt tokens by same-class distractor images, prompt distractors, and an auxiliary identity classifier.
- PerSegDelta: it amortizes these per-identity teachers. Specifically, it uses PCA to compress the offsets of ~1k training identities + an MLP on multi-layer DINOv2 CLS tokens regresses the coefficients.
- PerSeg-Adapt: it adds a residual MLP policy for video. It updates the coefficients every five frames from similarity-gated visual evidence.
The authors test PerSeg on a new 2×2 composite benchmark on PODS and DreamBooth. The experiments show: (a) hard negatives lift near-identity mIoU from 39.8 to 75.8; (b) PerSegDelta reaches 70.4 and beats mask-conditioned baselines.
Critical strengths
[S1]
I like the experiment parts where the authors clearly show that identity-level prompt personalization hinges on hard negatives + a cheap way to obtain them. Specifically, the proposed two kinds of negatives are complementary: image-space negatives add 20.9, prompt-space negatives with the identity head add 31.0, and together they add 39.9. This matters because it is the paper's most important takeaway (i.e., evaluations dominated by easy negatives can hide exactly the failure personalization must avoid).
[S2]
I also like the findings showing that mask-free & optimization-free inference can retain most of the teacher's performance. Specifically, the PCA trick is what makes this work. Regress the offset directly and it falls apart (24.9 IoU). Keep it in the span of the teacher offsets and it ends up only ~5 points behind the per-identity teacher (74.3 vs 79.2), even on the harder tiers. That's the real selling point: hand it one photo and you get a working prompt in one pass, with no mask and no per-subject tuning.
Critical weaknesses
[W1]
My major concern is that the evaluation does not show that learned identity offsets are what solves the task, and the headline comparisons are confounded. Specifically:
- The task reduces to instance selection. Specifically, SAM3 already segments the target well when there is no same-class competitor. The category prompt gets 21-24, and Direct and r = 512 give 25-32. This suggests a single-instance output, so the same-category tiers amount to four-way instance selection.
- What makes it even worse is that there is no comparison with an obvious mask-free baseline: use the SAM3 category prompt to get candidates, then pick the one whose DINOv2 descriptor (with same u(R)) or Qwen3-VL judgment best matches the reference.
- Moreover, identity-specific controls are missing. Specifically, a mean-offset (z = 0) control and a nearest-training-offset control would show whether the offset carries identity-specific information.
[W2]
Another concern is that several protocol choices could inflate the gains or misattribute them.
- First, the teacher ceiling may be closed-set. Specifically, PerSeg-DPT is tuned per test identity against same-class distractor images and the jointly learned prompts of other instances. If the composite distractors were in that negative pool, the headline near-identity result measures rejection of known distractors, not unseen ones.
- Second, r appears to be tuned on the test set, and the method is brittle to it.
- Lastly, "Mask-free" is weaker than framed, and the video comparisons are asymmetric. The reference must be object-centric, and in video it is cropped from the ground-truth annotation.
Questions for authors
[Q1]
I noticed that many DreamBooth categories contain a single subject, so where do its same-category distractors come from?
[Q2]
What exactly is the PerSegDelta reference for the composites and Table 14: the full image, or a crop derived from the ground-truth box or mask?
[Q3]
Do the reference and the target tile share capture context (background, session)? If so, can you show results where they don't, to rule out context matching by the image-level DINOv2 descriptor?
[Q4]
Where do the ground-truth masks for DreamBooth, DogFace, and GOT-10k come from, since none ships with masks? If they are SAM-derived, mIoU and BF1 will favor SAM3-based methods.
[Q5]
Why does FoundYou get a centroid point rather than the same box crop PerSeg receives?
6 comments
P3: mean-offset () control and a nearest-training-offset control would show whether the offset carries identity-specific information.
Reply: We thank the reviewer for suggesting these controls and have added them. Each control keeps the PerSegDelta pipeline and the category prompt fixed and replaces only the predicted identity offset: (i) mean offset (, the PCA mean), (ii) nearest training offset: the teacher offset of the most similar other training identity of the same category (nearest in predicted-coefficient space), (iii) shuffled: the offset PerSegDelta predicts for another identity of the same category, and (iv) random direction: a random coefficient vector with the norm of the predicted one. All rows use the same single unmasked reference and the same evaluation run (Table R9).
Table R9. Identity controls on the PODS composites, same-category tier (95% bootstrap CI over identities).
| Prompt | mIoU |
|---|---|
| mean offset () | 27.7 [23.2, 32.8] |
| random direction | 28.5 [23.9, 33.7] |
| shuffled same-class offset | 20.6 [15.6, 25.5] |
| nearest training offset | 31.1 [25.8, 36.8] |
| PerSegDelta (predicted offset) | 74.3 [68.3, 80.1] |
All controls fall to 20.6-31.1 mIoU, whereas the predicted offset reaches 74.3. Neither the offset of the nearest same-category training identity nor the offset of another instance of the same class transfers. The gain therefore comes from identity-specific information in the predicted offset, not from a class-level shift of the prompt. This is consistent with the retrieval ablation in Table 2, where removing the identity delta lowers mAP from 64.9 to 55.0.
The result also holds per subset: when targets are split by the PODS label of the target photo (pose, distractors, or both), PerSegDelta outperforms all controls in every subset, with the largest margin on new poses (72.6 vs. at most 36.1 mIoU).
Q3: Do the reference and the target tile share capture context (background, session)? If so, can you show results where they don't, to rule out context matching by the image-level DINOv2 descriptor?
Reply: Partly on PODS, and we thank the reviewer for raising this. Below, we report two evaluations in which the target's capture context differs from the reference's; they show that PerSegDelta does not rely on context matching. We also added a new figure in the revised paper (changes marked in blue) that shows reference and target photos are captured in different sessions and against different backgrounds. Further, two properties of the composite protocol also limit context matching. First, each composite cell is a crop at 1.8× the target box, so the target’s original background is not fully visible, while the reference is the full frame. Second, is computed from the reference alone and not from the composite.
PODS composites We split the same-category tier using PODS labels that change the capture context: a new pose, new scene distractors, or both (Table R10).
Table R10. PODS composites, same-category tier, subset of targets whose capture context differs from the reference.
| Method | Ref | Pose | Distractors | Pose + distractors |
|---|---|---|---|---|
| SAM3 + DINOv2 patch reranking | no mask | 37.2 | 45.2 | 20.8 |
| FoundYou | mask | 53.7 | 45.6 | 30.3 |
| PerSegDelta | no mask | 72.6 | 55.4 | 41.3 |
- Robust to new poses On targets that show the subject in a new pose, PerSegDelta reaches 72.6 mIoU, compared with 53.7 for FoundYou, even with a reference mask, and 37.2 for DINOv2 patch reranking. Methods that match the reference's appearance fail when the view changes, whereas PerSegDelta's identity prompt transfers across views.
- Scene distractors Cluttered target scenes lower every method, PerSegDelta included. PerSegDelta nonetheless remains the best method in the table on these targets: 55.4 mIoU with distractors (vs. at most 45.6) and 41.3 with pose changes and distractors (vs. at most 30.3).
ILIAS (Zero-shot evidence without shared context). On ILIAS (Table 12), each query is an object-centric photo on a plain background. The targets are the same objects in unrelated, cluttered scenes at other scales and positions, and PerSegDelta is never trained on ILIAS. From the unmasked query photo, PerSegDelta reaches 70.0 box IoU [66.9, 72.8], compared with 58.1 for FoundYou given the object box. Similarly, in ILIAS retrieval (Table 2), which uses the same setup, matching the background cannot retrieve them. In this setting, the image-level DINOv2 descriptor alone reaches only 27.8 mAP, whereas PerSegDelta reaches 64.9 mAP. Removing the predicted identity delta lowers this to 55.0 mAP. The identity information carried by the prompt therefore does not come from context shared between the reference and the target. We will add this breakdown to the paper and state the capture context of each benchmark.
P4: several protocol choices could inflate the gains or misattribute them.
Reply: The composite distractors are not a part of the teachers' training data. PerSeg-DPT teachers are trained only on the PODS training split, and their image distractors are drawn from that split as well, whereas every distractor in the composites is a test photo of another identity. There is therefore no overlap between the training images, including the negative pool, and the composite test images: every distractor photo is unseen. For PerSeg-DPT, the distractor identities are known. To measure the effect of this, we ran an ablation with two folds. We split the PODS test identities of every category into two folds and rebuilt the composites so that each distractor comes from the other fold (1,050 composites per tier, same tiers and scoring as Table 1). For each (category, fold) we then trained teachers with the original recipe. No composite distractor identity or images is therefore in their negative pool. Table R11 compares them. Removing the distractor identities from the negative pool lowers mIoU by only about 2.8 points (from 81.8 to 79.0 on same-category, 81.3 to 78.7 on similar, and 81.4 to 78.8 on near-identity), so knowing the distractor identities accounts for only a small part of the teacher's performance.
Table R11. Closed- vs. open-set teachers on PODS two-fold composites (distractors only from the other fold). mIoU, top-1; 95% bootstrap CI over identities.
| Prompt | Same category | Similar | Near-identity |
|---|---|---|---|
| PerSeg-DPT, (unseen distractor images, known identities) | 81.8 [78.2, 85.2] | 81.3 [77.6, 84.6] | 81.4 [77.7, 84.8] |
| PerSeg-DPT, (distractors never seen) | 79.0 [73.4, 84.4] | 78.7 [73.0, 83.7] | 78.8 [73.3, 83.6] |
P5: Second, appears to be tuned on the test set, and the method is brittle to it.
Reply: We selected based on an explained variance ratio of approximately 70%, aiming to balance expressivity and model capacity, without using any PODS test data. To further demonstrate that our choice is not dependent on test-set tuning, we conducted an additional experiment in which is selected using held-out validation data. Specifically, we train separate students for each and evaluate them on 30 DogFace validation identities excluded from training. We then select based on the validation mIoU, as reported in Table R12.
The validation performance generally improves with increasing , reaching its highest value of 90.5 at , which is also the default choice in the paper. Importantly, this validation-selected configuration achieves the best performance on both unseen DreamBooth evaluation settings (83.5 mIoU for same-category and 83.0 mIoU for near-category subjects). Moreover, performance is relatively stable for , suggesting that the method is not overly sensitive to the choice of within this range. These results demonstrate that can be selected reliably using held-out validation data, without relying on the PODS test set. We will clarify the validation-based selection procedure in the revised paper.
Table R12. Selecting on 30 held-out DogFace validation identities (mIoU on their test images). PerSegDelta is then evaluated on the DreamBooth composites (mIoU).
| Validation score | DreamBooth same-cat. | DreamBooth near | |
|---|---|---|---|
| 16 | 88.1 | 67.3 | 69.7 |
| 32 | 88.8 | 68.8 | 67.4 |
| 64 | 89.3 | 78.9 | 79.1 |
| 128 | 90.4 | 83.0 | 81.5 |
| 256 (selected) | 90.5 | 83.5 | 83.0 |
| 512 | 88.0 | 66.3 | 67.7 |
P6: Lastly, "mask-free" claim is weaker than framed, and the video comparisons are asymmetric. The reference must be object-centric, and in video it is cropped from the ground-truth annotation.
Reply: We agree that "mask-free" needs to be scoped precisely, and we will revise the wording accordingly (see reply to P5, 4aYQ). We will replace "mask-free" with "mask-free, given an object-centric reference and its category" in the abstract and introduction. The video experiments follow the label-propagation setting, in which the first-frame ground-truth mask is given as input and the task is to propagate it through the sequence. PerSegDelta uses this first-frame annotation only to crop its reference region.
The distinction from mask-conditioned methods still matters in practice. Table R13 compares FoundYou with its intended input, ground-truth reference masks, against the only RGB-only input it accepts, a box covering the whole reference image. Without a localized reference, FoundYou collapses: mask-conditioned methods cannot operate in the setting PerSegDelta targets without an additional localization step.
Table R13. FoundYou on Same-category tier.
| FoundYou reference | PODS | DreamBooth |
|---|---|---|
| Ground-truth masks | 59.0 | 81.6 |
| Whole-image (RGB only) | 1.2 | 6.9 |
We thank the reviewer for the detailed review and are glad that the reviewer found the role of hard negatives the paper's main takeaway and the PCA-based amortization its real selling point. We address the remaining concerns below.
P1: My major concern is that the evaluation does not show that learned identity offsets are what solves the task, and the headline comparisons are confounded. Specifically, the task reduces to instance selection. Specifically, SAM3 already segments the target well when there is no same-class competitor. The category prompt gets 21–24, and Direct and r=512 give 25–32. This suggests a single-instance output, so the same-category tiers amount to four-way instance selection.
Reply: We agree that, on the composites, SAM3's category candidates usually include the target (oracle selection: 91.7 mIoU on PODS and 98.0 on DreamBooth), so the benchmark mainly measures instance selection. Selecting the right instance among same-category ones is, however, the personalization problem itself, and SAM3's own scores do not solve it: its top-scoring candidate is the target in only 27% of the PODS composites. PerSegDelta makes this selection within a single SAM3 forward pass, without enumerating candidates or using a matching model. Its predicted prompt also serves as an identity embedding for retrieval (Table 2). We will add the reranking baseline (see P2) to Table 1.
P2: What makes it even worse is that there is no comparison with an obvious mask-free baseline: use the SAM3 category prompt to get candidates, then pick the one whose DINOv2 descriptor (with same u(R)) or Qwen3-VL judgment best matches the reference.
Reply: We thank the reviewer for suggesting this baseline, which we have added this result in Table R8. SAM3 with the category prompt proposes candidate masks (up to eight, score >0.05), and one candidate is selected by matching it to the reference. We report two matchers. Same u(R): each candidate's crop is encoded with u(⋅), the same DINOv2 encoder PerSegDelta applies to the reference (CLS tokens of blocks 2, 5, 8, and 11), and the candidate with the highest cosine similarity to u(R) is selected. This baseline differs from PerSegDelta only in how u(R) is used: nearest-candidate matching instead of the learned prompt. DINOv2 patches: for every patch inside the candidate mask, we take the cosine similarity to its nearest reference patch and average over the mask. In both cases, the reference is the same single unmasked image that PerSegDelta receives. Table R8 reports the same-category tier.
Table R8. Mask-free candidate reranking on the Table 1 composites, same-category tier (mIoU). All rows use one unmasked reference image.
On PODS, both rerankers remain well below PerSegDelta: reranking with the same descriptor u(R) reaches 61.5 mIoU, 12.8 below PerSegDelta (74.3), and patch reranking reaches 44.9, 29.4 below, selecting the correct cell in only 50% of the composites. Since the u(R) reranker uses exactly the information PerSegDelta receives, the gap shows that the learned prompt uses u(R) better than nearest-candidate matching. On DreamBooth, the three methods are on par overall: 85.0 for u(R) reranking and 82.5 for patch reranking, compared with 83.5 for PerSegDelta. As detailed in our reply to Q1(Table R15), only 54 of the 128 composites contain same-category distractors. So the overall DreamBooth score mostly reflects composites that do not test identity. Table R15 therefore reports every method on the 54 composites with at least one same-category distractor separately, where PerSegDelta leads both rerankers (84.3 vs. 78.7 for u(R) and 75.9 for patch reranking).