acceptodds
Under review as a conference paper at ICLR 2027

PerSeg: Learning Discriminative Prompts for Mask-Free Personalized Segmentation

Abstract

Prompt spaces in foundation models support semantic generalization, but can be insufficient for personalization, where a particular subject must be distinguished from visually similar instances of the same category. In this work, we study mask-free one-shot personalized segmentation: given a single RGB reference image of an unseen subject, without a reference mask or per-identity optimization, the goal is to segment that subject in a query image or video. We introduce , a framework that predicts personalized prompts from visual evidence while explicitly addressing fine-grained negative distractors. First, learns a personalized prompt teacher using positive supervision together with hard negatives, including same-class distractors. Second, amortizes this teacher by predicting an identity-specific offset from a semantic base prompt directly from the RGB reference image, enabling feed-forward personalization of unseen subjects. Third, predicts similarity-gated prompt updates to accommodate target appearance changes over time in video. To systematically evaluate identity discrimination, we introduce the benchmark, a controlled protocol in which the target remains fixed while distractors progress from different-category instances to visually similar and near-identity subjects. On near-identity distractors, negative-aware PerSeg-DPT improves mIoU from to over standard prompt tuning, while mask-free PerSegDelta achieves , approaching the teacher ceiling. PerSegDelta further obtains gains of points over the previous best method on the PODS and DreamBooth composite datasets. On ILIAS personalized retrieval, its prompt embeddings achieve a best mAP of , outperforming class-only and DINOv2 feature baselines. In video, PerSeg-Adapt improves performance by points over PerSegDelta on held-out YouTube-VOS validation. These results show that personalized prompts can be predicted from only a single RGB reference for unseen subjects, and effective personalization requires identity-specific visual representations and explicit supervision against hard negative distractors.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Segmentation, Personalization, One-Shot, Personalized Segmentation and Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pH8EOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets at mask-free one-shot personalized segmentation. The input is an unmasked RGB reference image of an unseen subject and its category name. The goal is to segment that specific instance in query images or video while rejecting same-category look-alikes.

The proposed PerSeg includes three stages:

  • PerSeg-DPT: it tunes per-identity prompt tokens by same-class distractor images, prompt distractors, and an auxiliary identity classifier.
  • PerSegDelta: it amortizes these per-identity teachers. Specifically, it uses PCA to compress the offsets of ~1k training identities + an MLP on multi-layer DINOv2 CLS tokens regresses the coefficients.
  • PerSeg-Adapt: it adds a residual MLP policy for video. It updates the coefficients every five frames from similarity-gated visual evidence.

The authors test PerSeg on a new 2×2 composite benchmark on PODS and DreamBooth. The experiments show: (a) hard negatives lift near-identity mIoU from 39.8 to 75.8; (b) PerSegDelta reaches 70.4 and beats mask-conditioned baselines.

Critical strengths

[S1]

I like the experiment parts where the authors clearly show that identity-level prompt personalization hinges on hard negatives + a cheap way to obtain them. Specifically, the proposed two kinds of negatives are complementary: image-space negatives add 20.9, prompt-space negatives with the identity head add 31.0, and together they add 39.9. This matters because it is the paper's most important takeaway (i.e., evaluations dominated by easy negatives can hide exactly the failure personalization must avoid).

[S2]

I also like the findings showing that mask-free & optimization-free inference can retain most of the teacher's performance. Specifically, the PCA trick is what makes this work. Regress the offset directly and it falls apart (24.9 IoU). Keep it in the span of the teacher offsets and it ends up only ~5 points behind the per-identity teacher (74.3 vs 79.2), even on the harder tiers. That's the real selling point: hand it one photo and you get a working prompt in one pass, with no mask and no per-subject tuning.

Critical weaknesses

[W1]

My major concern is that the evaluation does not show that learned identity offsets are what solves the task, and the headline comparisons are confounded. Specifically:

  • The task reduces to instance selection. Specifically, SAM3 already segments the target well when there is no same-class competitor. The category prompt gets 21-24, and Direct and r = 512 give 25-32. This suggests a single-instance output, so the same-category tiers amount to four-way instance selection.
  • What makes it even worse is that there is no comparison with an obvious mask-free baseline: use the SAM3 category prompt to get candidates, then pick the one whose DINOv2 descriptor (with same u(R)) or Qwen3-VL judgment best matches the reference.
  • Moreover, identity-specific controls are missing. Specifically, a mean-offset (z = 0) control and a nearest-training-offset control would show whether the offset carries identity-specific information.

[W2]

Another concern is that several protocol choices could inflate the gains or misattribute them.

  • First, the teacher ceiling may be closed-set. Specifically, PerSeg-DPT is tuned per test identity against same-class distractor images and the jointly learned prompts of other instances. If the composite distractors were in that negative pool, the headline near-identity result measures rejection of known distractors, not unseen ones.
  • Second, r appears to be tuned on the test set, and the method is brittle to it.
  • Lastly, "Mask-free" is weaker than framed, and the video comparisons are asymmetric. The reference must be object-centric, and in video it is cropped from the ground-truth annotation.
Questions for authors

[Q1]

I noticed that many DreamBooth categories contain a single subject, so where do its same-category distractors come from?

[Q2]

What exactly is the PerSegDelta reference for the composites and Table 14: the full image, or a crop derived from the ground-truth box or mask?

[Q3]

Do the reference and the target tile share capture context (background, session)? If so, can you show results where they don't, to rule out context matching by the image-level DINOv2 descriptor?

[Q4]

Where do the ground-truth masks for DreamBooth, DogFace, and GOT-10k come from, since none ships with masks? If they are SAM-derived, mIoU and BF1 will favor SAM3-based methods.

[Q5]

Why does FoundYou get a centroid point rather than the same box crop PerSeg receives?

6 comments

We thank the reviewer for the detailed review and are glad that the reviewer found the role of hard negatives the paper's main takeaway and the PCA-based amortization its real selling point. We address the remaining concerns below.

P1: My major concern is that the evaluation does not show that learned identity offsets are what solves the task, and the headline comparisons are confounded. Specifically, the task reduces to instance selection. Specifically, SAM3 already segments the target well when there is no same-class competitor. The category prompt gets 21–24, and Direct and r=512r=512 give 25–32. This suggests a single-instance output, so the same-category tiers amount to four-way instance selection.

Reply: We agree that, on the composites, SAM3's category candidates usually include the target (oracle selection: 91.7 mIoU on PODS and 98.0 on DreamBooth), so the benchmark mainly measures instance selection. Selecting the right instance among same-category ones is, however, the personalization problem itself, and SAM3's own scores do not solve it: its top-scoring candidate is the target in only 27% of the PODS composites. PerSegDelta makes this selection within a single SAM3 forward pass, without enumerating candidates or using a matching model. Its predicted prompt also serves as an identity embedding for retrieval (Table 2). We will add the reranking baseline (see P2) to Table 1.

P2: What makes it even worse is that there is no comparison with an obvious mask-free baseline: use the SAM3 category prompt to get candidates, then pick the one whose DINOv2 descriptor (with same u(R)u(R)) or Qwen3-VL judgment best matches the reference.

Reply: We thank the reviewer for suggesting this baseline, which we have added this result in Table R8. SAM3 with the category prompt proposes candidate masks (up to eight, score >0.05>0.05), and one candidate is selected by matching it to the reference. We report two matchers. Same u(R)u(R): each candidate's crop is encoded with u(⋅)u(\cdot), the same DINOv2 encoder PerSegDelta applies to the reference (CLS tokens of blocks 2, 5, 8, and 11), and the candidate with the highest cosine similarity to u(R)u(R) is selected. This baseline differs from PerSegDelta only in how u(R)u(R) is used: nearest-candidate matching instead of the learned prompt. DINOv2 patches: for every patch inside the candidate mask, we take the cosine similarity to its nearest reference patch and average over the mask. In both cases, the reference is the same single unmasked image that PerSegDelta receives. Table R8 reports the same-category tier.

Table R8. Mask-free candidate reranking on the Table 1 composites, same-category tier (mIoU). All rows use one unmasked reference image.

MethodPODS (1,050)DreamBooth (128)
SAM3 candidates + DINOv2 patch reranking44.982.5
SAM3 candidates + u(R)u(R) reranking (same descriptor)61.585.0
PerSegDelta74.383.5
Oracle candidate selection (upper bound)91.798.0

On PODS, both rerankers remain well below PerSegDelta: reranking with the same descriptor u(R)u(R) reaches 61.5 mIoU, 12.8 below PerSegDelta (74.3), and patch reranking reaches 44.9, 29.4 below, selecting the correct cell in only 50% of the composites. Since the u(R)u(R) reranker uses exactly the information PerSegDelta receives, the gap shows that the learned prompt uses u(R)u(R) better than nearest-candidate matching. On DreamBooth, the three methods are on par overall: 85.0 for u(R)u(R) reranking and 82.5 for patch reranking, compared with 83.5 for PerSegDelta. As detailed in our reply to Q1(Table R15), only 54 of the 128 composites contain same-category distractors. So the overall DreamBooth score mostly reflects composites that do not test identity. Table R15 therefore reports every method on the 54 composites with at least one same-category distractor separately, where PerSegDelta leads both rerankers (84.3 vs. 78.7 for u(R)u(R) and 75.9 for patch reranking).

P3: mean-offset (z=0z=0) control and a nearest-training-offset control would show whether the offset carries identity-specific information.

Reply: We thank the reviewer for suggesting these controls and have added them. Each control keeps the PerSegDelta pipeline and the category prompt pbase(c)p_{\text{base}}(c) fixed and replaces only the predicted identity offset: (i) mean offset (z=0z=0, the PCA mean), (ii) nearest training offset: the teacher offset of the most similar other training identity of the same category (nearest in predicted-coefficient space), (iii) shuffled: the offset PerSegDelta predicts for another identity of the same category, and (iv) random direction: a random coefficient vector with the norm of the predicted one. All rows use the same single unmasked reference and the same evaluation run (Table R9).

Table R9. Identity controls on the PODS composites, same-category tier (95% bootstrap CI over identities).

PromptmIoU
mean offset (z=0z=0)27.7 [23.2, 32.8]
random direction28.5 [23.9, 33.7]
shuffled same-class offset20.6 [15.6, 25.5]
nearest training offset31.1 [25.8, 36.8]
PerSegDelta (predicted offset)74.3 [68.3, 80.1]

All controls fall to 20.6-31.1 mIoU, whereas the predicted offset reaches 74.3. Neither the offset of the nearest same-category training identity nor the offset of another instance of the same class transfers. The gain therefore comes from identity-specific information in the predicted offset, not from a class-level shift of the prompt. This is consistent with the retrieval ablation in Table 2, where removing the identity delta lowers mAP from 64.9 to 55.0.

The result also holds per subset: when targets are split by the PODS label of the target photo (pose, distractors, or both), PerSegDelta outperforms all controls in every subset, with the largest margin on new poses (72.6 vs. at most 36.1 mIoU).

Q3: Do the reference and the target tile share capture context (background, session)? If so, can you show results where they don't, to rule out context matching by the image-level DINOv2 descriptor?

Reply: Partly on PODS, and we thank the reviewer for raising this. Below, we report two evaluations in which the target's capture context differs from the reference's; they show that PerSegDelta does not rely on context matching. We also added a new figure in the revised paper (changes marked in blue) that shows reference and target photos are captured in different sessions and against different backgrounds. Further, two properties of the composite protocol also limit context matching. First, each composite cell is a crop at 1.8× the target box, so the target’s original background is not fully visible, while the reference is the full frame. Second, u(R)u(R) is computed from the reference alone and not from the composite.

PODS composites We split the same-category tier using PODS labels that change the capture context: a new pose, new scene distractors, or both (Table R10).

Table R10. PODS composites, same-category tier, subset of targets whose capture context differs from the reference.

MethodRefPoseDistractorsPose + distractors
SAM3 + DINOv2 patch rerankingno mask37.245.220.8
FoundYoumask53.745.630.3
PerSegDeltano mask72.655.441.3
  • Robust to new poses On targets that show the subject in a new pose, PerSegDelta reaches 72.6 mIoU, compared with 53.7 for FoundYou, even with a reference mask, and 37.2 for DINOv2 patch reranking. Methods that match the reference's appearance fail when the view changes, whereas PerSegDelta's identity prompt transfers across views.
  • Scene distractors Cluttered target scenes lower every method, PerSegDelta included. PerSegDelta nonetheless remains the best method in the table on these targets: 55.4 mIoU with distractors (vs. at most 45.6) and 41.3 with pose changes and distractors (vs. at most 30.3).

ILIAS (Zero-shot evidence without shared context). On ILIAS (Table 12), each query is an object-centric photo on a plain background. The targets are the same objects in unrelated, cluttered scenes at other scales and positions, and PerSegDelta is never trained on ILIAS. From the unmasked query photo, PerSegDelta reaches 70.0 box IoU [66.9, 72.8], compared with 58.1 for FoundYou given the object box. Similarly, in ILIAS retrieval (Table 2), which uses the same setup, matching the background cannot retrieve them. In this setting, the image-level DINOv2 descriptor alone reaches only 27.8 mAP, whereas PerSegDelta reaches 64.9 mAP. Removing the predicted identity delta lowers this to 55.0 mAP. The identity information carried by the prompt therefore does not come from context shared between the reference and the target. We will add this breakdown to the paper and state the capture context of each benchmark.

P4: several protocol choices could inflate the gains or misattribute them.

Reply: The composite distractors are not a part of the teachers' training data. PerSeg-DPT teachers are trained only on the PODS training split, and their image distractors are drawn from that split as well, whereas every distractor in the composites is a test photo of another identity. There is therefore no overlap between the training images, including the negative pool, and the composite test images: every distractor photo is unseen. For PerSeg-DPT, the distractor identities are known. To measure the effect of this, we ran an ablation with two folds. We split the PODS test identities of every category into two folds and rebuilt the composites so that each distractor comes from the other fold (1,050 composites per tier, same tiers and scoring as Table 1). For each (category, fold) we then trained teachers with the original recipe. No composite distractor identity or images is therefore in their negative pool. Table R11 compares them. Removing the distractor identities from the negative pool lowers mIoU by only about 2.8 points (from 81.8 to 79.0 on same-category, 81.3 to 78.7 on similar, and 81.4 to 78.8 on near-identity), so knowing the distractor identities accounts for only a small part of the teacher's performance.

Table R11. Closed- vs. open-set teachers on PODS two-fold composites (distractors only from the other fold). mIoU, top-1; 95% bootstrap CI over identities.

PromptSame categorySimilarNear-identity
PerSeg-DPT, (unseen distractor images, known identities)81.8 [78.2, 85.2]81.3 [77.6, 84.6]81.4 [77.7, 84.8]
PerSeg-DPT, (distractors never seen)79.0 [73.4, 84.4]78.7 [73.0, 83.7]78.8 [73.3, 83.6]

P5: Second, rr appears to be tuned on the test set, and the method is brittle to it.

Reply: We selected rr based on an explained variance ratio of approximately 70%, aiming to balance expressivity and model capacity, without using any PODS test data. To further demonstrate that our choice is not dependent on test-set tuning, we conducted an additional experiment in which rr is selected using held-out validation data. Specifically, we train separate students for each r∈{16,32,64,128,256,512}r \in \{16, 32, 64, 128, 256, 512\} and evaluate them on 30 DogFace validation identities excluded from training. We then select rr based on the validation mIoU, as reported in Table R12.

The validation performance generally improves with increasing rr, reaching its highest value of 90.5 at r=256r=256, which is also the default choice in the paper. Importantly, this validation-selected configuration achieves the best performance on both unseen DreamBooth evaluation settings (83.5 mIoU for same-category and 83.0 mIoU for near-category subjects). Moreover, performance is relatively stable for r∈{128,256}r \in \{128, 256\}, suggesting that the method is not overly sensitive to the choice of rr within this range. These results demonstrate that rr can be selected reliably using held-out validation data, without relying on the PODS test set. We will clarify the validation-based selection procedure in the revised paper.

Table R12. Selecting rr on 30 held-out DogFace validation identities (mIoU on their test images). PerSegDelta is then evaluated on the DreamBooth composites (mIoU).

rrValidation scoreDreamBooth same-cat.DreamBooth near
1688.167.369.7
3288.868.867.4
6489.378.979.1
12890.483.081.5
256 (selected)90.583.583.0
51288.066.367.7

P6: Lastly, "mask-free" claim is weaker than framed, and the video comparisons are asymmetric. The reference must be object-centric, and in video it is cropped from the ground-truth annotation.

Reply: We agree that "mask-free" needs to be scoped precisely, and we will revise the wording accordingly (see reply to P5, 4aYQ). We will replace "mask-free" with "mask-free, given an object-centric reference and its category" in the abstract and introduction. The video experiments follow the label-propagation setting, in which the first-frame ground-truth mask is given as input and the task is to propagate it through the sequence. PerSegDelta uses this first-frame annotation only to crop its reference region.

The distinction from mask-conditioned methods still matters in practice. Table R13 compares FoundYou with its intended input, ground-truth reference masks, against the only RGB-only input it accepts, a box covering the whole reference image. Without a localized reference, FoundYou collapses: mask-conditioned methods cannot operate in the setting PerSegDelta targets without an additional localization step.

Table R13. FoundYou on Same-category tier.

FoundYou referencePODSDreamBooth
Ground-truth masks59.081.6
Whole-image (RGB only)1.26.9
Reviewer_4aYQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies one-shot personalized segmentation, where the target instance must be distinguished from near-identical same-category objects. It conditions a frozen SAM3 on a prompt p(s)=pbase(c)+Δ(s)p(s) = p_{\text{base}}(c) + \Delta(s), where the offset Δ(s)∈RK×D\Delta(s) \in \mathbb{R}^{K \times D} is first learned per instance with image and prompt distractors (teacher), and then predicted from DINOv2 features of the reference image in a PCA subspace, Δ^(s)=δˉ+Wz^(s)\hat{\Delta}(s) = \bar{\delta} + W \hat{z}(s), trained on 10001000 identities, so that no mask or test-time optimization is needed (student). On 2×22 \times 2 composite benchmarks built from PODS and DreamBooth, the student reaches 74.3/83.574.3 / 83.5 mIoU versus 59.0/81.659.0 / 81.6 for FoundYou, with additional results on ILIAS retrieval, YouTube-VOS, and DAVIS-person.

Critical strengths

The negative-difficulty analysis is the most useful part of the paper. It shows directly that ordinary prompt tuning learns "a tote bag" rather than "this tote bag": its mIoU drops from 81.381.3 to 39.839.8 as negatives become closer to the target, whereas PerSegDelta degrades much less (87.2→70.487.2 \rightarrow 70.4). The ablation in Table 3 attributes this to the image distractors, prompt distractors, and identity loss (39.26→79.1839.26 \rightarrow 79.18). The amortized student is also a sensible design: predicting coefficients in a PCA subspace of teacher offsets removes per-instance optimization at test time, at a cost of only about 55 points relative to the teacher on the hardest tier.

Critical weaknesses
  1. The headline comparison mixes training regimes, and the gains shrink outside it. PerSegDelta is trained on teacher offsets of 10001000 identities, including PODS training identities, and is evaluated on held-out PODS identities in the same composite format. GF-SAM, FoundYou, PerSAM, and PDM are training-free and never see this data. The +15.3+15.3 mIoU on PODS shrinks to +1.9+1.9 on DreamBooth, the only cross-dataset image benchmark. On YouTube-VOS, PerSeg-Adapt beats plain SAM3 box tracking by 1.21.2 Dice (72.472.4 vs. 71.271.2), and static PerSegDelta (70.070.0) is below it. Retrieval shows the same pattern: the abstract highlights mAP (64.964.9 vs. 59.559.5), but FoundYou is clearly better on MRR (84.184.1 vs. 77.777.7) and R@1 (81.381.3 vs. 69.469.4). No confidence intervals are reported, and DreamBooth has only 128128 composites, so I cannot tell whether the out-of-distribution gains are reliable.

  2. The benchmark is a narrow test of personalization, and "mask-free" needs qualification. Test images are 2×22 \times 2 grids of 1.8×1.8\times-scaled object crops, with negatives selected by RGB similarity of 64×6464 \times 64 crops. This isolates identity discrimination cleanly, but removes clutter, scale variation, occlusion, and the localization part of the problem. In addition, the method requires the category name cc for pbase(c)p_{\text{base}}(c) (predicted by Qwen3-VL in retrieval), and Appendix H states that it assumes the target dominates the reference image. The DAVIS-person comparison gives FoundYou only a single frame-0 centroid point while PerSegDelta receives the reference region, which is not a like-for-like comparison.

Questions for authors
  1. Can you train at least one baseline under the same supervision, e.g., a SAM3 prompt learner or FoundYou fine-tuned on the same 10001000 training identities? This would separate the method's design from the extra training data.
  2. Do you have results on natural multi-instance images rather than composites, even on a small set?
  3. Please report variance over composite resamplings, especially for DreamBooth and YouTube-VOS, where margins are small.
  4. In the segmentation experiments, how much does performance drop when the category name cc is predicted by the VLM instead of given?
  5. What is the total compute for producing 10001000 teacher offsets at roughly 40004000 iterations each?
5 comments

We thank the reviewer for the detailed review and are glad that the reviewer found the negative-difficulty analysis the most useful part of the paper and the amortized student a sensible design. We address the remaining concerns below.

P1: The headline comparison mixes training regimes, and the gains shrink outside it. PerSegDelta is trained on teacher offsets of about 1,000 identities, including PODS training identities, and is evaluated on held-out PODS identities in the same composite format. GF-SAM, FoundYou, PerSAM, and PDM are training-free and never see this data.

Q1: Can you train at least one baseline under the same supervision, e.g., a SAM3 prompt learner or FoundYou fine-tuned on the same training identities? This would separate the method's design from the extra training data.

Reply: We would like to clarify that the baselines are not supervision-free. GF-SAM, FoundYou, PerSAM, and PDM require a ground-truth segmentation mask of every reference image at test time. PerSegDelta instead receives a single unmasked RGB reference. The comparison therefore trades a one-time training cost for per-reference mask annotation at test time, rather than adding supervision only on our side. The masks are essential to these methods: given only the RGB reference, FoundYou drops to 1.2 mIoU on PODS and 6.9 on DreamBooth (Table R13 in our reply to Reviewer pH8E, P6).

We also trained a SAM3 prompt learner under exactly the same supervision: single-stage students that map the reference to a prompt offset and are trained end to end with the teacher's loss on the same identities, without the teacher stage (Table R1 in our reply to Reviewer 8shH, P1). On PODS, the best one matches PerSegDelta trained on the same identities (74.3 mIoU). On the zero-shot DreamBooth composites it falls to 19.7, while PerSegDelta keeps 83.5. The training data alone therefore does not explain the gain on unseen subjects; the method's design does. Moreover, FoundYou is not training-free: it is trained on UnED, which contains 2.8M training images of 349k classes across 8 domains, far more than the 1,000 identities used to train PerSegDelta. Despite this, on the zero-shot benchmarks, it outperforms FoundYou on ILIAS in-the-wild photos (70.0 vs. 58.1 box IoU, Table R5) and on PerMIRS (63.8 vs. 61.7 J, Table R4). We could not fine-tune FoundYou, because its public repository provides inference code only, and reimplementing its training was not feasible within the rebuttal period with compute constraints.

P2: The +15.3+15.3 mIoU on PODS shrinks to +1.9+1.9 on DreamBooth, the only cross-dataset image benchmark. On YouTube-VOS, PerSeg-Adapt beats plain SAM3 box tracking by 1.21.2 Dice (72.472.4 vs. 71.271.2), and static PerSegDelta (70.070.0) is below it. No confidence intervals are reported, and DreamBooth has only 128 composites, so I cannot tell whether the out-of-distribution gains are reliable.

Q3: Please report variance over composite resamplings, especially for DreamBooth and YouTube-VOS, where margins are small.

Reply: We now report 95% bootstrap confidence intervals over identities, and we repeated the composite construction with four additional random seeds. Composite resampling has little effect: across the five seeds, PerSegDelta obtains 74.3, 72.6, 73.1, 71.9, and 73.6 mIoU (per-seed CI about ±6\pm6). The intervals are much wider on DreamBooth, which has only 30 subjects; for example, the DINOv2 patch reranker (Table R8) obtains 82.5 [75.0, 88.1] there.

We also report an additional zero-shot image benchmark: on 1,378 in-the-wild ILIAS photos of 291 objects never seen in training, PerSegDelta reaches 70.0 box IoU [66.9, 72.8] from an unmasked reference, compared with 58.1 [54.8, 61.1] for FoundYou given the object box (Table R5; see our reply to Q2).

On YouTube-VOS, with 95% bootstrap intervals over videos, PerSegDelta obtains 70.0 [68.4, 72.1] Dice and PerSeg-Adapt 72.4 [70.6, 74.3].

P3: Retrieval shows the same pattern: the abstract highlights mAP (64.9 vs. 59.5), but FoundYou is clearly better on MRR (84.1 vs. 77.7) and R@1 (81.3 vs. 69.4).

Reply: We agree that the retrieval claim was stated too broadly, and we will revise it. The purpose of Table 2 is to show that the predicted prompt carries identity information even though it is never trained for retrieval: removing the identity delta reduces mAP from 64.9 to 55.0 and R@1 from 69.4 to 46.3. While FoundYou is better at the top of the ranking (MRR and R@1), PerSegDelta ranks the full set of same-identity images more consistently (mAP and R@10). Moreover, FoundYou requires a reference mask at query time, whereas PerSegDelta uses a single RGB image. We will rewrite the abstract and Section 4.2 to state exactly this.

Table R3. ILIAS personalized retrieval.

MethodmAPMRRR@1R@10
FoundYou (reference mask)59.584.181.390.8
Qwen3-VL + PerSegDelta (RGB only)64.977.769.494.0
Qwen3-VL class prompt only55.055.146.376.2

P4: The benchmark is a narrow test of personalization. This isolates identity discrimination cleanly, but removes clutter, scale variation, occlusion, and the localization part of the problem. Q2: Do you have results on natural multi-instance images rather than composites, even on a small set?

Reply: Beyond the composites, the paper already reports the results below, and we have added two new benchmarks; we will reference all of them in Section 4:

  • Non-composite images (Appendix E, Table 14). One reference per identity, evaluated on the ordinary PODS and DogFace test images, with no distractor grid, crop, or rescaling. The model therefore also has to localize the target in the full image, including the cluttered PODS test images. PerSegDelta, without a reference mask, obtains 86.9 mIoU on PODS and 93.0 on DogFace, compared with 81.8 and 91.9 for FoundYou with a reference mask (Table R6).
  • Natural multi-instance video. YouTube-VOS (Table 4) evaluates 11,292 held-out frames, including crowded scenes with many same-category instances (bird flock, ice hockey, multi-basketball, multi-soccer, pedestrians, skiing).
  • Qualitative natural scenes (Appendix F, Fig. 7). The reference van is segmented in a street scene with several white cars, and only the matching puppy is segmented among three puppies of the same breed.
  • New: zero-shot generalization to ILIAS in-the-wild photos (Table R5). We segment the 291 ILIAS test objects in all 1,378 of their in-the-wild photos, from the single object-centric query photo (no reference mask). The photos are cluttered natural scenes in which the object appears at other scales and positions, so the model has to localize it, not only recognize it. ILIAS provides boxes only, so we score the bounding box of the predicted mask against the ground-truth box. PerSegDelta is not trained on ILIAS. From the unmasked query photo alone, it reaches 70.0 box IoU, compared with 58.1 for FoundYou given the object box and 56.5 for DINOv2 patch reranking of SAM3 candidates.
  • New: zero-shot natural multi-instance scenes (PerMIRS, Table R4). PerMIRS contains 432 episodes from BURST videos in which several instances of the same category appear together. Given only an unmasked crop of the frame-0 reference, PerSegDelta obtains 63.8 J, compared with 61.7 for FoundYou given a frame-0 box and 34.3 for DINOv2 patch reranking of SAM3 candidates.

We agree that the composites remain the most controlled test of identity discrimination. We have now added Table R10 (reply to pH8E) for pose, distractors control that also contain clutter, scale and occlusion (See Figure 4 of paper). We will add these results to the paper.

Table R4. Zero-shot PerMIRS: natural multi-instance scenes. J (region IoU), averaged over query frames, with 95% bootstrap CI over identities.

MethodJ
SAM3 candidates + DINOv2 patch reranking34.3 [29.1, 39.5]
FoundYou61.7 [56.5, 66.5]
PerSegDelta63.8 [58.7, 68.6]

Table R5. Zero-shot generalization to ILIAS in-the-wild photos. IoU between the predicted mask's bounding box and the ground-truth box, with 95% bootstrap CI.

MethodRefBox IoU
SAM3 candidates + DINOv2 patch rerankingquery photo, no mask56.5 [53.3, 59.6]
FoundYouquery + object box58.1 [54.8, 61.1]
FoundYouquery, whole-image box26.4 [23.7, 29.6]
PerSegDeltaquery, no mask70.0 [66.9, 72.8]
Best SAM3 candidate (oracle)—79.4 [77.0, 81.9]

Table R6. Non-composite, single-reference personalized segmentation

MethodRef maskPODS mIoUDogFace mIoU
FoundYouYes81.891.9
PerSAM-FYes68.587.8
PDMYes62.275.2
PerSegDeltaNo86.993.0

P5: "Mask-free" needs qualification. In addition, the method requires the category name cc for pbase(c)p_{\text{base}}(c) (predicted by Qwen3-VL in retrieval), and Appendix H states that it assumes the target dominates the reference image.

Reply: We agree, and we will scope the claim as described in our reply to Reviewer pH8E (P6): PerSegDelta is mask-free given an object-centric reference (standard for personalization) and its category, and we will move the Appendix H assumptions into the main text. Table R13 shows why the mask-free distinction still matters: FoundYou relies on masked references in Table 1, and given only the RGB reference it drops to 1.2 mIoU on PODS and 6.9 on DreamBooth.

Q4: In the segmentation experiments, how much does performance drop when the category name cc is predicted by the VLM instead of given?

Reply: We report t his in Table R7. Qwen3-VL-8B names the category from the same unmasked reference image PerSegDelta receives. The predicted name replaces cc in pbase(c)p_{\text{base}}(c); nothing else changes (Table R7).

Table R7. Given vs. VLM-predicted category name cc on the composites (mIoU; 95% bootstrap CI over identities).

MethodccPODS same cat.PODS near-id.DreamBooth same cat.DreamBooth near-id.
PerSegDeltagiven74.3 [68.3, 80.1]70.4 [63.6, 76.7]83.5 [79.1, 87.2]83.0 [79.1, 87.5]
PerSegDeltapredicted71.4 [64.8, 76.9]67.2 [60.5, 73.5]86.3 [71.9, 98.9]87.8 [73.5, 99.0]

Given the VLM's prediction, PerSegDelta drops by 2.9 mIoU on PODS, and even improves on DreamBooth by 2.8 mIoU. The category name therefore can be inferred from a VLM and does not need to be supplied by a user.

P6: The DAVIS-person comparison gives FoundYou only a single frame-0 centroid point while PerSegDelta receives the reference region, which is not a like-for-like comparison.

Reply: Both methods use the first-frame box to identify the subject: PerSegDelta only crops its reference with it and receives no spatial prompt, so FoundYou was provided with the smallest spatial prompt it accepts, a single centroid point. Given the same first-frame box, FoundYou improves to 53.7 J&F but stays below PerSegDelta (60.1) and PerSeg-Adapt (60.9); see Table R14 in our reply to Reviewer pH8E, Q5. We will add this row to Table 6, labeled with the reference input of each method, and revise the DAVIS-person discussion accordingly.

Table R14. DAVIS-person.

MethodFrame-0 inputJFJ&F
FoundYoucentroid point (as in paper)30.225.828.0
FoundYouground-truth box53.054.453.7
GF-SAMground-truth mask46.352.449.4
PerSegDeltaRGB crop from the box, no spatial prompt61.458.960.1
PerSeg-AdaptRGB crop from the box + online updates61.460.560.9

Q5: What is the total compute for producing the 1,000 teacher offsets at roughly 4,000 iterations each?

Reply: Each teacher is trained for 4,000 iterations per identity with 0.41 GPU-hours per identity on an RTX 3090. For 1000 teacher offsets, it requires about 410 GPU hours. This is a one-time cost for training. At test time, PerSegDelta handles a new identity with one DINOv2 and one SAM3 forward pass, with no teacher training and no optimization. We will add these numbers to the paper.

Reviewer_8shHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerSeg proposes a mask-free approach for one-shot personalised segmentation. The paper argues that prompt spaces learned by foundation models like SAM are insufficient to represent personalised instances from visual evidence. To address this, the authors propose PerSeg, which predicts personalised prompts capable of distinguishing specific instances from other instances of the same category while also differentiating between categories.

The authors introduce PerSeg-DPT, a teacher based on prompt learning that learns to distinguish personalised instances from other instances. PerSegDelta then learns to predict the difference between the PerSeg-DPT teacher prompts and base category prompts. A coefficient predictor uses DINOv2 features to predict this difference in a lower-dimensional space.

The paper is well written and provides clear motivation for the proposed method. The results show improvements over existing state-of-the-art methods on the PODS and DreamBooth datasets.

Critical strengths
  1. The paper is well written and provides clear motivation for the proposed method. The problem of distinguishing personalised instances from other instances of the same category is well motivated. The proposed methodological components are also interesting, particularly the use of PerSeg-DPT to learn discriminative personalised prompts and PerSegDelta to predict personalised prompts using a coefficient predictor in a lower-dimensional space. The overall framework is well motivated, with the different components addressing different aspects of personalised segmentation.

  2. The proposed method demonstrates improvements over existing state-of-the-art methods on the PODS and DreamBooth datasets. These improvements support the effectiveness of the proposed approach in learning personalised prompts capable of distinguishing specific instances from others within the same category.

Critical weaknesses
  1. The paper first trains PerSeg-DPT to learn personalised teacher prompts and then trains PerSegDelta to predict the difference between the teacher prompts and base category prompts. It is unclear whether this two-stage procedure is necessary or whether the same objective could be achieved using a single network. Additionally, the authors use DINOv2 features for the coefficient predictor, but do not investigate whether SAM3 encoder features could be used instead, potentially avoiding the need for an additional model.

  2. The ablation studies primarily focus on the losses used in training PerSeg-DPT, while experiments investigating the necessity of PerSeg-DPT and the individual components of PerSegDelta are missing. These missing ablations make it difficult to assess the necessity and contribution of the different methodological components.

  3. The evaluation of PerSeg-Adapt on video segmentation lacks comparisons with other state-of-the-art personalised video segmentation methods. Including such comparisons would help establish how well the proposed video adaptation approach performs relative to existing methods.

Questions for authors

Please see weaknesses

4 comments

We thank the reviewer for the detailed review and are glad that the reviewer found the paper well written, the problem well motivated, and the DPT and PerSegDelta components interesting. We address the remaining concerns below.

P1: The paper first trains PerSeg-DPT to learn personalised teacher prompts and then trains PerSegDelta to predict the difference between the teacher prompts and base category prompts. It is unclear whether this two-stage procedure is necessary or whether the same objective could be achieved using a single network.

Reply: We trained single-stage students that map the same DINOv2 reference descriptor straight to the prompt offset and are optimized end to end through the frozen SAM3 with the teacher's loss, on the same training identities and images as PerSegDelta, but without teacher prompts. We tried three heads: a direct offset regressor, a CoCoOp-style head (shared context plus a per-image bias), and a head that predicts coefficients in PerSegDelta's PCA basis. Table R1 compares them with PerSegDelta trained on the same identities.

Table R1. Is the teacher stage needed? All students use the same DINOv2 descriptor, training identities and images; only the two-stage row regresses teacher offsets. Composite mIoU (top-1); 95% bootstrap CI on the same-category tier. DreamBooth is zero-shot (no subject seen in training).

MethodPODS same-cat.PODS nearDreamBooth (zero-shot) same-cat.DreamBooth (zero-shot) near
PerSegDelta (two-stage)74.370.483.583.0
Single-stage, direct offset74.269.619.725.7
Single-stage, CoCoOp-style65.062.032.532.2
Single-stage, PCA-basis coefficients58.754.642.541.9

On PODS, which evaluates new images of training identities, the best single-stage student matches PerSegDelta (74.3 mIoU on the same-category tier). On the zero-shot DreamBooth composites, every single-stage student performance drops to 19.7-42.5, while PerSegDelta keeps 83.5. A single network can fit the training identities, but its offsets do not transfer to new subjects. Even the head restricted to PerSegDelta's PCA subspace trails PerSegDelta by 41.0 mIoU, which suggests that regressing onto per-identity teacher offsets, rather than the subspace alone, is what lets the student generalize. The two-stage procedure is therefore needed for better generalization.

P2: Additionally, the authors use DINOv2 features for the coefficient predictor, but do not investigate whether SAM3 encoder features could be used instead, potentially avoiding the need for an additional model.

Reply: We retrained PerSegDelta with the reference descriptor taken from the frozen SAM3 image encoder (the spatially pooled FPN features of the same backbone, at its native 1008-pixel input) and kept everything else fixed: the same teachers and training identities, predictor, r=256r = 256, and schedule. Table R2 compares it with the DINOv2 descriptor of the paper and a small ViT trained from scratch.

Table R2. Reference encoder of PerSegDelta. All students are trained on the same identities and teachers with r=256r = 256; only the descriptor differs. Composite mIoU (top-1); 95% bootstrap CI on the same-category tier.

Reference descriptorPODS same-cat.PODS nearDreamBooth same-cat.DreamBooth near
DINOv2 B/14, blocks 2, 5, 8, 1174.3 [68.3, 80.1]70.483.5 [79.1, 87.2]83.0
Small ViT65.7 [59.6, 71.1]65.451.9 [36.0, 65.7]52.8
SAM3 image encoder (pooled FPN)30.7 [25.5, 36.5]25.454.2 [40.2, 69.0]52.1

SAM3's own features do not work as the reference descriptor: on PODS they reach only 30.7 mIoU, 43.6 below DINOv2, and on DreamBooth they trail DINOv2 by 29.3 mIoU on the same-category tier. A likely reason is that SAM3's encoder is more semantic, trained for promptable, category-level segmentation as compared to instance-level self-supervised DINOv2 features. The additional encoder for instance features is therefore needed and it runs once, on the reference image only with minimal cost.

P3: The ablation studies primarily focus on the losses used in training PerSeg-DPT, while experiments investigating the necessity of PerSeg-DPT and the individual components of PerSegDelta are missing. These missing ablations make it difficult to assess the necessity and contribution of the different methodological components.

Reply: We added these ablations. Necessity of PerSeg-DPT: Table R1 (reply to P1) trains single-stage students with the teacher's loss on the same data but without the teacher stage. They match PerSegDelta on PODS but fall to 19.7-42.5 mIoU on the zero-shot DreamBooth composites (PerSegDelta 83.5). Components of PerSegDelta: the predicted offset is identity-specific (Table R9, reply to Reviewer pH8E, P3), the DINOv2 reference encoder cannot be replaced by SAM3's own features (Table R2, P2), and the number of principal components rr is selected on held-out validation identities (Table R12, reply to Reviewer pH8E, P5).

P4: The evaluation of PerSeg-Adapt on video segmentation lacks comparisons with other state-of-the-art personalised video segmentation methods. Including such comparisons would help establish how well the proposed video adaptation approach performs relative to existing methods.

Reply: We thank the reviewer for the suggestion. The paper already compares with FoundYou, the strongest prior personalized segmentation method (Table 1), and we now report it on DAVIS-person given the same first-frame box as PerSegDelta. In addition, we have added GF-SAM, a recent one-shot personalized segmentation method, which receives the first frame together with the subject's ground-truth mask as its reference and segments the subject in each later frame (Table R14). PerSegDelta (60.1 J&F) and PerSeg-Adapt (60.9) outperform both FoundYou with the box (53.7) and GF-SAM with the ground-truth mask (49.4), although they use the first-frame box only to crop their reference and receive no mask or spatial prompt. To our knowledge, there is no dedicated personalized video segmentation method with public code that segments a subject from a single unmasked reference; the closest alternatives are general video object segmentation trackers, which we compare with on YouTube-VOS (SAM3 box tracking, paper Table 4). We will add both baselines to Table 6.

Thanks for your detailed response.

I am satisfied with the rebuttal and I would like to keep my score.

Dear Reviewer 8shH,

Thank you for the rebuttal response. We are glad that your concerns are addressed. We will include the revisions in the final manuscript.

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (Summary) to the AC and All Reviewers

We thank all the reviewers and the AC for their time and constructive feedback. Reviewers found the paper well written and well motivated (8shH), the negative-difficulty analysis its most useful takeaway (4aYQ, pH8E), and the amortized PCA-based student a sensible design and real selling point (4aYQ, pH8E). We ran the requested experiments (Tables R1–R16); the key findings are below.

1. The learned identity offset drives the gains.

  • Identity controls (pH8E): replacing the predicted offset (mean, random, shuffled, or nearest training offset) drops PODS mIoU from 74.3 to 20.6-31.1 (R9).
  • Mask-free reranking (pH8E): reranking SAM3 candidates with the same DINOv2 descriptor reaches 61.5 vs. 74.3 on PODS (R8). On the 54 DreamBooth composites with same-category distractors, PerSegDelta leads (84.3 vs. 78.7 reranking; 82.9 for FoundYou with a mask; R15).
  • Same supervision, no teacher (8shH, 4aYQ): single-stage prompt learners match PerSegDelta on PODS (74.2 vs. 74.3) but fall to 19.7–42.5 on zero-shot DreamBooth vs. 83.5 (R1). The design, not the training data, explains the gain.
  • Open-set teachers (pH8E): teachers whose negatives exclude every composite distractor identity and image still reach 78.7-79.0 mIoU, only ~2.8 below the teachers who saw the distractor identities (not their test images) (R11). Their discrimination therefore comes from discriminative training against same-class negatives, not from knowing the distractor identities.

2. Gains hold beyond composites, zero-shot (4aYQ, pH8E).

  • ILIAS in-the-wild photos (new): 70.0 vs. 58.1 box IoU for FoundYou given the box (R5).
  • PerMIRS multi-instance scenes (new): 63.8 vs. 61.7 J for FoundYou (R4).
  • Non-composite PODS / DogFace: 86.9 / 93.0 mIoU without a mask vs. 81.8 / 91.9 for FoundYou with one (R6).
  • New-pose PODS targets: 72.6 vs. 53.7 for FoundYou, ruling out context matching (R10).

3. Robustness and protocol.

  • We report 95% bootstrap CIs, five composite seeds give 71.9–74.3 mIoU (4aYQ).
  • rr chosen on held-out validation identities selects r=256r=256, the paper's default (R12, pH8E).
  • A VLM-predicted category name decreases performance by 2.9 mIoU on PODS (within CIs), and improves slightly on DreamBooth (R7, 4aYQ).
  • SAM3's own features cannot replace DINOv2 (30.7 vs. 74.3; R2, 8shH).
  • DAVIS-person: FoundYou with the same box gets 53.7 J&F and GF-SAM with the mask 49.4, vs. 60.1 / 60.9 for PerSegDelta / PerSeg-Adapt (R14).
  • Teacher training is a one-time 410 GPU-hours for 1000 teacher offsets; a new identity needs only a single DINOV2 and SAM3 forward pass (4aYQ).

4. Revisions

  • "Mask-free" scoped to "given an object-centric reference and its category" which is standard for personalization; These assumptions will be clearly stated in the main text.
  • Retrieval claim corrected: we lead on mAP / R@10, FoundYou on MRR / R@1. The retrieval table shows the predicted prompt carries identity information even though it is never trained for retrieval.
  • Reranking baselines, identity controls, and the 54-composite DreamBooth split will be added to Table 1.
  • Table 14, PerMIRS, and ILIAS results moved to the main paper.
  • FoundYou-with-box and GF-SAM added to Table 6.
  • New ablations, CIs, and compute added to the appendix.

We believe these address the reviewers' concerns, and we would be glad to clarify anything further during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.