- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new evaluation framework to address the issue in PAR where heterogeneous label-imbalance interventions (e.g., reweighting) paired with a small set of numerical inference thresholds cause the score meaning of the commonly used metric mA to shift under different operating policies, making cross-method comparisons unfair. The authors argue that directly comparing native mA has fairness flaws. To address this, the authors develop a decision-coordinate framework that derives objective-implied coordinates (κobj) and estimates trained predictors' score-space policy coordinates (κest) through validation-set isotonic calibration, to perform objective normalization and realized-policy normalization, respectively. This framework aligns the decision boundaries of different methods to the common balanced-accuracy decision target (κ=0), thereby eliminating the interference caused by operating-policy differences and achieving a fair comparison at the evaluation level.
Critical strengths
- This paper identifies a practically relevant and often overlooked evaluation issue in PAR: heterogeneous label-imbalance interventions can alter the meaning of prediction scores, causing native mA comparisons to involve different operating policies. To address this issue, the authors develop a decision-coordinate framework with objective normalization and realized-policy normalization, providing a more comparable evaluation of PAR methods under a common balanced-accuracy decision target.
- This paper provides a systematic theoretical analysis supported by extensive empirical evaluations. The authors establish the relationship between training objectives, inference thresholds, and operating-policy coordinates, and conduct experiments on three widely used PAR benchmarks (PETA, PA100K, and RAP) using ResNet-50 and ViT-B/16 backbones. The evaluation covers controlled label-imbalance interventions, published PAR methods, comparisons across evaluation views, and sensitivity analyses of calibration choices and validation data, providing empirical support for the main findings.
Critical weaknesses
Experiments show that normalized method rankings may change with the calibration method. For the highly class-imbalanced PAR task, rare attributes have few positive samples, which further increases the uncertainty in calibration and threshold estimation. The authors need to quantify this uncertainty more explicitly, especially when the performance gaps between methods are very small.
Questions for authors
- The paper mainly reports changes in mA gaps after normalization, but their implications remain unclear. What do changes in mA after normalization reveal about the sources of a method's performance improvements?
- The authors are encouraged to analyze how normalization affects different label-imbalance interventions, such as resampling and loss reweighting [1], to further demonstrate the framework's interpretive value.
- The authors could select a representative PAR study (e.g., [2]) and re-evaluate four or five methods from its original comparisons, reporting both native and normalized mA.
- Further analysis of whether the original performance advantages and method rankings persist after normalization would help demonstrate the framework's practical value in reassessing existing PAR studies.
[1] Pedestrian Attribute Recognition as Label-balanced Multi-label Learning [2]Pedestrian Attribute Recognition via CLIP-based Prompt Vision-Language Fusion
5 comments
Response to Reviewer xLTa (2/3)
[Q1] What do changes in mA gaps reveal about the sources of performance gains?
A higher Native mA shows an advantage for the complete PAR system under its documented inference rule. Normalization lets us examine how much of that advantage persists when the predictors stay fixed and both methods use decision rules that target balanced accuracy.
-
A change in the gap measures how much the comparison depends on the decision rule. We hold each predictor’s scores fixed and construct Obj. and Est. thresholds for the common balanced-accuracy target (). Obj. derives thresholds from identifiable loss or sampling components, while Est. uses the score-to-posterior relation estimated from validation scores and labels. Each predictor’s ranking of images for each attribute stays the same across views.
-
Operating-policy misalignment can contribute to a Native lead. Table 2 gives a concrete reason why the Native gap can shrink. At the same threshold of 0.5, inverse-prior reweighting places the objective-implied boundary at the mA target (), whereas ordinary BCE requires stronger positive evidence for attributes with (). On RAP with ResNet-50, moving from Native to Est. raises ERM’s mA by about 9.6 pp and that of by only 0.8 pp (Table 3). The gap falls from +8.99 to +0.12 pp (Table 13). ERM therefore closes much of the gap using scores it had already learned, supporting operating-policy misalignment as a contributor to the Native advantage.
-
A method’s advantage can also persist or grow under the common target. Normalization also helps interpret comparisons between methods that use the same imbalance intervention. L2L and PIL both use -weighted BCE in their classification components and a Native threshold of 0.5 (Table 10). Yet L2L’s lead on RAP grows from 0.37 pp under Native to 2.28 pp under Obj. and 2.69 pp under Est. (Table 3). In this case, the advantage persists under both rules targeting balanced accuracy, even though its size depends on the decision rule. Attributing its advantage to a particular training component would require a method-specific ablation.
[Q2] How does normalization affect resampling and loss reweighting?
For the resampling and loss-reweighting methods in Table 3, Obj. changes the Native threshold when the decision boundary implied by the training rule differs from the mA target. Est. uses the score-to-posterior relation estimated for each trained predictor, so it can change the threshold even when the sampling or weighting rule already aligns the objective-implied boundary with that target.
-
Under Obj., balanced feature resampling and inverse-prior reweighting behave alike. Both give , so FRDL and retain their Native thresholds on all three datasets and both backbones. For exponential reweighting (), the Obj. threshold depends on the attribute’s training prior. Moving from Native to Obj. raises its ResNet-50 mA by 4.2 pp on PETA, 4.8 pp on PA100K, and 6.7 pp on RAP.
-
Under Est., FRDL overtakes on both PETA and PA100K with ResNet-50. On PETA, for example, leads FRDL under both Native and Obj. (85.97% vs. 85.83%). Under Est., FRDL reaches 87.53% and reaches 87.33%. For , the ranking changes one step earlier: on both datasets, it moves from the lowest Native mA of the three methods to the highest mean mA under Obj. and retains that lead under Est.
Response to Reviewer xLTa (3/3)
[Q3] Re-evaluation of methods compared in the PromptPAR study [2].
Following your suggestion, we revisited the comparisons in the PromptPAR study, cited as [2] in your review.
[2] Xiao Wang, Jiandong Jin, Chenglong Li, Jin Tang, Cheng Zhang, and Wei Wang. "Pedestrian Attribute Recognition via CLIP-Based Prompt Vision-Language Fusion." IEEE Transactions on Circuits and Systems for Video Technology, 35(1), 148–161, 2025.
We selected four method families represented in Tables I–II of [2]: VAC, SSC, PARFormer, and PromptPAR. We report five configurations because we evaluate both the released PromptPAR checkpoint (*PromptPAR) and our retrained PromptPAR. For the retrained models, only the training split contributed to supervised parameter updates, and the validation split was used for calibration. The official PromptPAR code defaults to trainval, which includes our validation images, although the exact training and selection history of the released weights is not fully documented (Section 4.1 and Appendices D.1 and D.2.2). This difference from the default training protocol can affect how closely our retrained PromptPAR matches the Native mA reported in [2].
For both VAC and SSC, we used a Native probability threshold of 0.5 and derived the Obj. thresholds from each method's weighted BCE classification component. Let denote the positive rate of attribute in the training split. Both methods weight positive examples by . VAC weights negative examples by , while SSC gives them unit weight. The table below reports test mA (%) for all five configurations under Native, Obj., and Est. (Isotonic).
On PA100K, our Native results reproduce the direction of the SSC–VAC comparison in [2], while both normalized rules reverse it. Table I of [2] reports higher mA for SSCsoft than for VAC (81.87% vs. 79.16%). In our Native evaluation, SSC leads VAC by 1.94 pp. Under Obj., VAC leads SSC by 1.42 pp, and under Est. (Isotonic) it leads by 1.67 pp. The same reversal occurs on PETA and RAP in our evaluation. On PA100K, the released *PromptPAR retains the highest mA among the five configurations under all three rules.
| Method | PETA Native | PETA Obj. | PETA Est. | PA100K Native | PA100K Obj. | PA100K Est. | RAP Native | RAP Obj. | RAP Est. |
|---|---|---|---|---|---|---|---|---|---|
| VAC | 83.57 ± 0.42 | 88.32 ± 0.22 | 88.33 ± 0.23 | 78.92 ± 0.20 | 85.02 ± 0.25 | 86.71 ± 0.40 | 78.58 ± 0.25 | 85.85 ± 0.05 | 86.27 ± 0.25 |
| SSC | 85.79 ± 0.29 | 86.63 ± 0.36 | 86.68 ± 0.11 | 80.86 ± 0.15 | 83.60 ± 1.12 | 85.04 ± 0.39 | 81.76 ± 0.37 | 84.61 ± 0.21 | 85.32 ± 0.21 |
| PARFormer-B | 87.99 ± 0.15 | 88.82 ± 0.26 | 89.24 ± 0.23 | 81.85 ± 0.34 | 85.06 ± 0.75 | 86.83 ± 0.57 | 82.73 ± 0.33 | 86.01 ± 0.80 | 86.49 ± 0.53 |
| *PromptPAR (released) | 87.79 | 88.99 | 88.02 | 87.23 | 89.89 | 90.10 | 85.42 | 89.06 | 88.83 |
| PromptPAR (retrained) | 86.85 ± 0.12 | 88.99 ± 0.12 | 88.92 ± 0.13 | 85.35 ± 0.34 | 88.32 ± 0.58 | 88.88 ± 0.26 | 84.22 ± 0.09 | 87.81 ± 0.08 | 87.77 ± 0.15 |
[Q4] Further analysis of whether Native performance advantages and method rankings persist after normalization.
We analyzed the Native leader and all six pairwise orderings by reported mA within each architecture group on PETA, PA100K, and RAP. We used the four CNN methods in Table 3. For the Transformer group, we used HAP, PARFormer-B, ReSAGE-PAR, and the released *PromptPAR checkpoint, which represents PromptPAR in our original comparison.
-
The Native leader changes in five of the six settings under Est. (Isotonic). In the CNN group, FRDL+GOAT moves from first on all three datasets to second on PETA and PA100K and third on RAP. In the Transformer group, HAP overtakes PARFormer-B on PETA and released *PromptPAR on PA100K. Released *PromptPAR retains its lead on RAP.
-
Most pairwise orderings persist even though the top rank often changes. Of the 36 within-group comparisons, 20 retain their Native ordering under Est. (Isotonic) and 16 reverse. On PA100K, for example, FRDL+GOAT has two nearly equal Native leads: 3.05 pp over ReSAGE-PAR and 3.14 pp over L2L. Under Est. (Isotonic), it trails ReSAGE-PAR by 0.69 pp but still leads L2L by 1.69 pp. Obj. gives the same two orderings. Table 13 reports paired intervals for ReSAGE-PAR minus FRDL+GOAT on all three datasets. On PA100K, the 95% interval is [−3.41, −2.54] pp under Native and [+0.28, +1.30] pp under Est. (Isotonic). We provide the individual run results for PETA and RAP in our response to
Reviewer@CX2L [W3/Q3].
These comparisons concern rankings by mA. We also examined which orderings persist for instance-level metrics (response to Reviewer@Po7R [Q1]) and how Est. mA rankings compare with macro-AUROC rankings (Appendix E.5, Table 16; response to Reviewer@JJ8Q [Q3]).
Response to Reviewer xLTa (1/3)
We appreciate your suggestions to quantify threshold uncertainty for rare attributes and to revisit a published PAR comparison to see which performance advantages persist under normalization. Below, we address the concern raised under Critical Weaknesses and then respond to your four questions in order.
We quantified the effect of validation sampling on two quantities: the Est. threshold for each attribute and its per-attribute test mA (BAj in Section 3.1). For each quantity, we measured the standard deviation (SD) across validation refits. To compare thresholds across attributes and predictors, we expressed each Est. threshold as its percentile among the corresponding scores on the full validation set.
We chose ERM and RWinv on ResNet-50 because their Est. (Isotonic) mA gap on RAP is only 0.12 pp and its paired interval includes zero (Table 13). Across all three datasets, we used the three fixed predictors per method from Table 3 and refitted both calibrators on the same 1,000 validation bootstrap samples per dataset, keeping the training priors and test images fixed. For each attribute, we computed both SDs over supported refits and averaged them across the three predictors. We have added this analysis to the revised paper (
Appendix E.6, Table 17). The table below reports the group medians as SD of the Est. threshold percentile / SD of per-attribute test mA, both in pp.In all 12 dataset–method–calibrator combinations, the group with a validation positive rate below 5% had the highest median SD for both the Est. threshold and per-attribute test mA. The median SD of per-attribute test mA decreased from [0%, 5%) to [5%, 25%) and again to [25%, 50%) in every combination. For example, on PETA, the median threshold SD for ERM under Est. (Isotonic) was 3.75, 1.97, 1.92, and 2.15 pp across the four groups in order of increasing validation positive rate. The corresponding median SDs of per-attribute test mA were 1.24, 0.36, 0.22, and 0.32 pp. One possible explanation is that, for rare attributes, the calibration fit relies on relatively few positive validation images. Which of those images are repeated or omitted in a bootstrap sample can shift where the fitted curve crosses the fixed training prior, changing the Est. threshold and, in turn, the per-attribute test mA.